Mercurial > cpython-lru_cache-weakref

annotate Objects/unicodeobject.c @ 49955:878d40468c4e legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #4384: Added logging integration with warnings module using captureWarnings(). This change includes a NullHandler which does nothing; it will be of use to library developers who want to avoid the "No handlers could be found for logger XXX" message which can appear if the library user doesn't configure logging.
author Vinay Sajip <vinay_sajip@yahoo.co.uk>
date Wed, 03 Dec 2008 23:22:58 +0000
parents 3c7eef187bea
children e673632e1885
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1 /*
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3 Unicode implementation based on original code by Fredrik Lundh,
12646
17d1a3f1f085 Minimal change so I can add the rest of MAL's checkin message:
Fred Drake <fdrake@acm.org>
parents: 12645
diff changeset
4 modified by Marc-Andre Lemburg <mal@lemburg.com> according to the
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 Unicode Integration Proposal (see file Misc/unicode.txt).
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7 Major speed upgrades to the method implementations at the Reykjavik
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
8 NeedForSpeed sprint, by Fredrik Lundh and Andrew Dalke.
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
9
13828
254f8b7c4a83 Changing the CNRI copyright notice according to CNRI's instructions.
Guido van Rossum <guido@python.org>
parents: 13761
diff changeset
10 Copyright (c) Corporation for National Research Initiatives.
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
11
16029
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
12 --------------------------------------------------------------------
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
13 The original string type implementation is:
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
14
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
15 Copyright (c) 1999 by Secret Labs AB
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
16 Copyright (c) 1999 by Fredrik Lundh
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
17
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
18 By obtaining, using, and/or copying this software and/or its
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
19 associated documentation, you agree that you have read, understood,
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
20 and will comply with the following terms and conditions:
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
21
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
22 Permission to use, copy, modify, and distribute this software and its
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
23 associated documentation for any purpose and without fee is hereby
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
24 granted, provided that the above copyright notice appears in all
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
25 copies, and that both that copyright notice and this permission notice
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
26 appear in supporting documentation, and that the name of Secret Labs
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
27 AB or the author not be used in advertising or publicity pertaining to
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
28 distribution of the software without specific, written prior
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
29 permission.
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
30
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
31 SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
32 THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
33 FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR BE LIABLE FOR
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
34 ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
35 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
36 ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
37 OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
38 --------------------------------------------------------------------
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
39
55d89a8fe963 refactored the unicodeobject/ucnhash interface, to hide the
Fredrik Lundh <fredrik@pythonware.com>
parents: 15952
diff changeset
40 */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
41
37702
9b22d8b7bcaa Make Py_BuildValue, PyObject_CallFunction and
Martin v. Löwis <martin@v.loewis.de>
parents: 37668
diff changeset
42 #define PY_SSIZE_T_CLEAN
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 #include "Python.h"
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
44
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
45 #include "unicodeobject.h"
13209
b11e0a898de0 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13196
diff changeset
46 #include "ucnhash.h"
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
47
24071
8142f8d668a0 Patch #569753: Remove support for WIN16.
Martin v. Löwis <martin@v.loewis.de>
parents: 23837
diff changeset
48 #ifdef MS_WINDOWS
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
49 #include <windows.h>
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
50 #endif
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
51
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
52 /* Limit for the Unicode object free list */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
53
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
54 #define PyUnicode_MAXFREELIST 1024
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
55
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
56 /* Limit for the Unicode object free list stay alive optimization.
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
57
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 The implementation will keep allocated Unicode memory intact for
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 all objects on the free list having a size less than this
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
60 limit. This reduces malloc() overhead for small Unicode objects.
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
61
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
62 At worst this will result in PyUnicode_MAXFREELIST *
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
63 (sizeof(PyUnicodeObject) + KEEPALIVE_SIZE_LIMIT +
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
64 malloc()-overhead) bytes of unused garbage.
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
65
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
66 Setting the limit to 0 effectively turns the feature off.
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
67
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
68 Note: This is an experimental feature ! If you get core dumps when
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
69 using Unicode objects, turn this feature off.
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
70
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
71 */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
72
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
73 #define KEEPALIVE_SIZE_LIMIT 9
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
74
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
75 /* Endianness switches; defaults to little endian */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
76
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
77 #ifdef WORDS_BIGENDIAN
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
78 # define BYTEORDER_IS_BIG_ENDIAN
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
79 #else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
80 # define BYTEORDER_IS_LITTLE_ENDIAN
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
81 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
82
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
83 /* --- Globals ------------------------------------------------------------
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
84
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
85 The globals are initialized by the _PyUnicode_Init() API and should
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
86 not be used before calling that API.
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
87
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
88 */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
89
37655
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
90
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
91 #ifdef __cplusplus
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
92 extern "C" {
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
93 #endif
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
94
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
95 /* Free list for Unicode objects */
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
96 static PyUnicodeObject *free_list;
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
97 static int numfree;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
98
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
99 /* The empty Unicode object is shared to improve performance. */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
100 static PyUnicodeObject *unicode_empty;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
101
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
102 /* Single character Unicode strings in the Latin-1 range are being
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
103 shared as well. */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
104 static PyUnicodeObject *unicode_latin1[256];
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
105
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
106 /* Default encoding to use and assume when NULL is passed as encoding
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
107 parameter; it is initialized by _PyUnicode_Init().
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
108
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
109 Always use the PyUnicode_SetDefaultEncoding() and
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
110 PyUnicode_GetDefaultEncoding() APIs to access this global.
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
111
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
112 */
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
113 static char unicode_default_encoding[100];
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
114
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
115 /* Fast detection of the most frequent whitespace characters */
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
116 const unsigned char _Py_ascii_whitespace[] = {
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
117 0, 0, 0, 0, 0, 0, 0, 0,
49386
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
118 /* case 0x0009: * HORIZONTAL TABULATION */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
119 /* case 0x000A: * LINE FEED */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
120 /* case 0x000B: * VERTICAL TABULATION */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
121 /* case 0x000C: * FORM FEED */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
122 /* case 0x000D: * CARRIAGE RETURN */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
123 0, 1, 1, 1, 1, 1, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
124 0, 0, 0, 0, 0, 0, 0, 0,
49386
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
125 /* case 0x001C: * FILE SEPARATOR */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
126 /* case 0x001D: * GROUP SEPARATOR */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
127 /* case 0x001E: * RECORD SEPARATOR */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
128 /* case 0x001F: * UNIT SEPARATOR */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
129 0, 0, 0, 0, 1, 1, 1, 1,
49386
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
130 /* case 0x0020: * SPACE */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
131 1, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
132 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
133 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
134 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
135
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
136 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
137 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
138 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
139 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
140 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
141 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
142 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
143 0, 0, 0, 0, 0, 0, 0, 0
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
144 };
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
145
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
146 /* Same for linebreaks */
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
147 static unsigned char ascii_linebreak[] = {
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
148 0, 0, 0, 0, 0, 0, 0, 0,
49386
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
149 /* 0x000A, * LINE FEED */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
150 /* 0x000D, * CARRIAGE RETURN */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
151 0, 0, 1, 0, 0, 1, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
152 0, 0, 0, 0, 0, 0, 0, 0,
49386
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
153 /* 0x001C, * FILE SEPARATOR */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
154 /* 0x001D, * GROUP SEPARATOR */
642f20b22577 Merged revisions 66748 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 49115
diff changeset
155 /* 0x001E, * RECORD SEPARATOR */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
156 0, 0, 0, 0, 1, 1, 1, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
157 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
158 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
159 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
160 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
161
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
162 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
163 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
164 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
165 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
166 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
167 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
168 0, 0, 0, 0, 0, 0, 0, 0,
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
169 0, 0, 0, 0, 0, 0, 0, 0
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
170 };
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
171
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
172
18298
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
173 Py_UNICODE
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
174 PyUnicode_GetMax(void)
18298
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
175 {
18324
3ba7a358ce02 use Py_UNICODE_WIDE instead of USE_UCS4_STORAGE and Py_UNICODE_SIZE
Fredrik Lundh <fredrik@pythonware.com>
parents: 18298
diff changeset
176 #ifdef Py_UNICODE_WIDE
18298
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
177 return 0x10FFFF;
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
178 #else
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
179 /* This is actually an illegal character, so it should
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
180 not be passed to unichr. */
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
181 return 0xFFFF;
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
182 #endif
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
183 }
982163c4e9d2 Encode surrogates in UTF-8 even for a wide Py_UNICODE.
Martin v. Löwis <martin@v.loewis.de>
parents: 18295
diff changeset
184
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
185 /* --- Bloom Filters ----------------------------------------------------- */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
186
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
187 /* stuff to implement simple "bloom filters" for Unicode characters.
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
188 to keep things simple, we use a single bitmask, using the least 5
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
189 bits from each unicode characters as the bit index. */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
190
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
191 /* the linebreak mask is set up by Unicode_Init below */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
192
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
193 #define BLOOM_MASK unsigned long
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
194
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
195 static BLOOM_MASK bloom_linebreak;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
196
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
197 #define BLOOM(mask, ch) ((mask & (1 << ((ch) & 0x1F))))
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
198
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
199 #define BLOOM_LINEBREAK(ch) \
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
200 ((ch) < 128U ? ascii_linebreak[(ch)] : \
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
201 (BLOOM(bloom_linebreak, (ch)) && Py_UNICODE_ISLINEBREAK(ch)))
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
202
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
203 Py_LOCAL_INLINE(BLOOM_MASK) make_bloom_mask(Py_UNICODE* ptr, Py_ssize_t len)
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
204 {
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
205 /* calculate simple bloom-style bitmask for a given unicode string */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
206
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
207 long mask;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
208 Py_ssize_t i;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
209
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
210 mask = 0;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
211 for (i = 0; i < len; i++)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
212 mask |= (1 << (ptr[i] & 0x1F));
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
213
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
214 return mask;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
215 }
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
216
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
217 Py_LOCAL_INLINE(int) unicode_member(Py_UNICODE chr, Py_UNICODE* set, Py_ssize_t setlen)
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
218 {
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
219 Py_ssize_t i;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
220
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
221 for (i = 0; i < setlen; i++)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
222 if (set[i] == chr)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
223 return 1;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
224
38270
1dc5657f1745 return 0 on misses, not -1.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38257
diff changeset
225 return 0;
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
226 }
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
227
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
228 #define BLOOM_MEMBER(mask, chr, set, setlen)\
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
229 BLOOM(mask, chr) && unicode_member(chr, set, setlen)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
230
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
231 /* --- Unicode Object ----------------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
232
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
233 static
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
234 int unicode_resize(register PyUnicodeObject *unicode,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
235 Py_ssize_t length)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
236 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
237 void *oldstr;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
238
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
239 /* Shortcut if there's nothing much to do. */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
240 if (unicode->length == length)
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
241 goto reset;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
242
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
243 /* Resizing shared object (unicode_empty or single character
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
244 objects) in-place is not allowed. Use PyUnicode_Resize()
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
245 instead ! */
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
246
37661
b1c6a3864b1e Revert 34153: Py_UNICODE should not be signed.
Martin v. Löwis <martin@v.loewis.de>
parents: 37655
diff changeset
247 if (unicode == unicode_empty ||
b1c6a3864b1e Revert 34153: Py_UNICODE should not be signed.
Martin v. Löwis <martin@v.loewis.de>
parents: 37655
diff changeset
248 (unicode->length == 1 &&
b1c6a3864b1e Revert 34153: Py_UNICODE should not be signed.
Martin v. Löwis <martin@v.loewis.de>
parents: 37655
diff changeset
249 unicode->str[0] < 256U &&
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
250 unicode_latin1[unicode->str[0]] == unicode)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
251 PyErr_SetString(PyExc_SystemError,
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
252 "can't resize shared unicode objects");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
253 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
254 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
255
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
256 /* We allocate one more byte to make sure the string is Ux0000 terminated.
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
257 The overallocation is also used by fastsearch, which assumes that it's
38405
3128cfada6b4 Comment typo
Andrew M. Kuchling <amk@amk.ca>
parents: 38404
diff changeset
258 safe to look at str[length] (without making any assumptions about what
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
259 it contains). */
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
260
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
261 oldstr = unicode->str;
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
262 unicode->str = PyObject_REALLOC(unicode->str,
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
263 sizeof(Py_UNICODE) * (length + 1));
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
264 if (!unicode->str) {
37596
1b078493d538 More low-hanging fruit. Still need to re-arrange some code (or find a better
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37306
diff changeset
265 unicode->str = (Py_UNICODE *)oldstr;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
266 PyErr_NoMemory();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
267 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
268 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
269 unicode->str[length] = 0;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
270 unicode->length = length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
271
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
272 reset:
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
273 /* Reset the object caches */
13837
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
274 if (unicode->defenc) {
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
275 Py_DECREF(unicode->defenc);
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
276 unicode->defenc = NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
277 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
278 unicode->hash = -1;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
279
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
280 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
281 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
282
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
283 /* We allocate one more byte to make sure the string is
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
284 Ux0000 terminated -- XXX is this needed ?
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
285
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
286 XXX This allocator could further be enhanced by assuring that the
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
287 free list never reduces its size below 1.
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
288
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
289 */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
290
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
291 static
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
292 PyUnicodeObject *_PyUnicode_New(Py_ssize_t length)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
293 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
294 register PyUnicodeObject *unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
295
38426
9f3375e33db9 fixed typo
Andrew Dalke <dalke@dalkescientific.com>
parents: 38424
diff changeset
296 /* Optimization for empty strings */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
297 if (length == 0 && unicode_empty != NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
298 Py_INCREF(unicode_empty);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
299 return unicode_empty;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
300 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
301
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
302 /* Ensure we won't overflow the size. */
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
303 if (length > ((PY_SSIZE_T_MAX / sizeof(Py_UNICODE)) - 1)) {
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
304 return (PyUnicodeObject *)PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
305 }
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
306
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
307 /* Unicode freelist & memory allocation */
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
308 if (free_list) {
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
309 unicode = free_list;
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
310 free_list = *(PyUnicodeObject **)unicode;
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
311 numfree--;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
312 if (unicode->str) {
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
313 /* Keep-Alive optimization: we only upsize the buffer,
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
314 never downsize it. */
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
315 if ((unicode->length < length) &&
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
316 unicode_resize(unicode, length) < 0) {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
317 PyObject_DEL(unicode->str);
48117
a94f136b5283 Correct a crash when two successive unicode allocations fail with a MemoryError:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 48114
diff changeset
318 unicode->str = NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
319 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
320 }
18120
c29f3135a20c Fix a mis-indentation in _PyUnicode_New() that caused me to stare at
Guido van Rossum <guido@python.org>
parents: 18051
diff changeset
321 else {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
322 size_t new_size = sizeof(Py_UNICODE) * ((size_t)length + 1);
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
323 unicode->str = (Py_UNICODE*) PyObject_MALLOC(new_size);
18120
c29f3135a20c Fix a mis-indentation in _PyUnicode_New() that caused me to stare at
Guido van Rossum <guido@python.org>
parents: 18051
diff changeset
324 }
c29f3135a20c Fix a mis-indentation in _PyUnicode_New() that caused me to stare at
Guido van Rossum <guido@python.org>
parents: 18051
diff changeset
325 PyObject_INIT(unicode, &PyUnicode_Type);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
326 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
327 else {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
328 size_t new_size;
22981
dc1fb19d990d Remove PyMalloc_*.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 22938
diff changeset
329 unicode = PyObject_New(PyUnicodeObject, &PyUnicode_Type);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
330 if (unicode == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
331 return NULL;
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
332 new_size = sizeof(Py_UNICODE) * ((size_t)length + 1);
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
333 unicode->str = (Py_UNICODE*) PyObject_MALLOC(new_size);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
334 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
335
12511
8747f1c8b5fb Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12369
diff changeset
336 if (!unicode->str) {
8747f1c8b5fb Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12369
diff changeset
337 PyErr_NoMemory();
12025
7994d9db7f4d On 17-Mar-2000, Marc-Andre Lemburg said:
Barry Warsaw <barry@python.org>
parents: 11997
diff changeset
338 goto onError;
12511
8747f1c8b5fb Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12369
diff changeset
339 }
30439
30e495efe90a Double-fix of crash in Unicode freelist handling.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30431
diff changeset
340 /* Initialize the first element to guard against cases where
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
341 * the caller fails before initializing str -- unicode_resize()
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
342 * reads str[0], and the Keep-Alive optimization can keep memory
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
343 * allocated for str alive across a call to unicode_dealloc(unicode).
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
344 * We don't want unicode_resize to read uninitialized memory in
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
345 * that case.
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
346 */
30439
30e495efe90a Double-fix of crash in Unicode freelist handling.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30431
diff changeset
347 unicode->str[0] = 0;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
348 unicode->str[length] = 0;
37668
4342e441b11c Remove another INT_MAX limitation
Martin v. Löwis <martin@v.loewis.de>
parents: 37662
diff changeset
349 unicode->length = length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
350 unicode->hash = -1;
13837
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
351 unicode->defenc = NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
352 return unicode;
12025
7994d9db7f4d On 17-Mar-2000, Marc-Andre Lemburg said:
Barry Warsaw <barry@python.org>
parents: 11997
diff changeset
353
7994d9db7f4d On 17-Mar-2000, Marc-Andre Lemburg said:
Barry Warsaw <barry@python.org>
parents: 11997
diff changeset
354 onError:
48117
a94f136b5283 Correct a crash when two successive unicode allocations fail with a MemoryError:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 48114
diff changeset
355 /* XXX UNREF/NEWREF interface should be more symmetrical */
a94f136b5283 Correct a crash when two successive unicode allocations fail with a MemoryError:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 48114
diff changeset
356 _Py_DEC_REFTOTAL;
12025
7994d9db7f4d On 17-Mar-2000, Marc-Andre Lemburg said:
Barry Warsaw <barry@python.org>
parents: 11997
diff changeset
357 _Py_ForgetReference((PyObject *)unicode);
22981
dc1fb19d990d Remove PyMalloc_*.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 22938
diff changeset
358 PyObject_Del(unicode);
12025
7994d9db7f4d On 17-Mar-2000, Marc-Andre Lemburg said:
Barry Warsaw <barry@python.org>
parents: 11997
diff changeset
359 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
360 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
361
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
362 static
20277
2983b54256c7 Enable GC for new-style instances. This touches lots of files, since
Guido van Rossum <guido@python.org>
parents: 20018
diff changeset
363 void unicode_dealloc(register PyUnicodeObject *unicode)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
364 {
21262
e8fea95a82a4 Fix for #489669 (Neil Norwitz): memory leak in test_descr (unicode).
Guido van Rossum <guido@python.org>
parents: 21082
diff changeset
365 if (PyUnicode_CheckExact(unicode) &&
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
366 numfree < PyUnicode_MAXFREELIST) {
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
367 /* Keep-Alive optimization */
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
368 if (unicode->length >= KEEPALIVE_SIZE_LIMIT) {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
369 PyObject_DEL(unicode->str);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
370 unicode->str = NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
371 unicode->length = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
372 }
13837
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
373 if (unicode->defenc) {
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
374 Py_DECREF(unicode->defenc);
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
375 unicode->defenc = NULL;
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
376 }
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
377 /* Add to free list */
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
378 *(PyUnicodeObject **)unicode = free_list;
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
379 free_list = unicode;
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
380 numfree++;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
381 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
382 else {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
383 PyObject_DEL(unicode->str);
13837
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
384 Py_XDECREF(unicode->defenc);
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
385 Py_TYPE(unicode)->tp_free((PyObject *)unicode);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
386 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
387 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
388
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
389 int PyUnicode_Resize(PyObject **unicode, Py_ssize_t length)
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
390 {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
391 register PyUnicodeObject *v;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
392
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
393 /* Argument checks */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
394 if (unicode == NULL) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
395 PyErr_BadInternalCall();
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
396 return -1;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
397 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
398 v = (PyUnicodeObject *)*unicode;
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
399 if (v == NULL || !PyUnicode_Check(v) || Py_REFCNT(v) != 1 || length < 0) {
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
400 PyErr_BadInternalCall();
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
401 return -1;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
402 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
403
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
404 /* Resizing unicode_empty and single character objects is not
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
405 possible since these are being shared. We simply return a fresh
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
406 copy with the same Unicode content. */
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
407 if (v->length != length &&
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
408 (v == unicode_empty || v->length == 1)) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
409 PyUnicodeObject *w = _PyUnicode_New(length);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
410 if (w == NULL)
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
411 return -1;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
412 Py_UNICODE_COPY(w->str, v->str,
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
413 length < v->length ? length : v->length);
28242
fe1d571c3f80 Sf patch #700047: unicode object leaks refcount on resizing
Raymond Hettinger <python@rcn.com>
parents: 27826
diff changeset
414 Py_DECREF(*unicode);
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
415 *unicode = (PyObject *)w;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
416 return 0;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
417 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
418
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
419 /* Note that we don't have to modify *unicode for unshared Unicode
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
420 objects, since we can modify them in-place. */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
421 return unicode_resize(v, length);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
422 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
423
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
424 /* Internal API for use in unicodeobject.c only ! */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
425 #define _PyUnicode_Resize(unicodevar, length) \
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
426 PyUnicode_Resize(((PyObject **)(unicodevar)), length)
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
427
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
428 PyObject *PyUnicode_FromUnicode(const Py_UNICODE *u,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
429 Py_ssize_t size)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
430 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
431 PyUnicodeObject *unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
432
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
433 /* If the Unicode data is known at construction time, we can apply
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
434 some optimizations which share commonly used objects. */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
435 if (u != NULL) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
436
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
437 /* Optimization for empty strings */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
438 if (size == 0 && unicode_empty != NULL) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
439 Py_INCREF(unicode_empty);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
440 return (PyObject *)unicode_empty;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
441 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
442
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
443 /* Single character Unicode objects in the Latin-1 range are
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
444 shared when using this constructor */
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
445 if (size == 1 && *u < 256) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
446 unicode = unicode_latin1[*u];
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
447 if (!unicode) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
448 unicode = _PyUnicode_New(1);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
449 if (!unicode)
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
450 return NULL;
18051
e897550b4748 Fixes [ #430986 ] Buglet in PyUnicode_FromUnicode.
Marc-André Lemburg <mal@egenix.com>
parents: 17995
diff changeset
451 unicode->str[0] = *u;
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
452 unicode_latin1[*u] = unicode;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
453 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
454 Py_INCREF(unicode);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
455 return (PyObject *)unicode;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
456 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
457 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
458
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
459 unicode = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
460 if (!unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
461 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
462
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
463 /* Copy the Unicode data into the new object */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
464 if (u != NULL)
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
465 Py_UNICODE_COPY(unicode->str, u, size);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
466
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
467 return (PyObject *)unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
468 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
469
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
470 PyObject *PyUnicode_FromStringAndSize(const char *u, Py_ssize_t size)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
471 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
472 PyUnicodeObject *unicode;
46383
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
473
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
474 if (size < 0) {
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
475 PyErr_SetString(PyExc_SystemError,
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
476 "Negative size passed to PyUnicode_FromStringAndSize");
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
477 return NULL;
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
478 }
0b55875c1c9c Raise SystemError when size < 0 is passed into PyString_FromStringAndSize,
Gregory P. Smith <greg@mad-scientist.com>
parents: 46333
diff changeset
479
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
480 /* If the Unicode data is known at construction time, we can apply
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
481 some optimizations which share commonly used objects.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
482 Also, this means the input must be UTF-8, so fall back to the
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
483 UTF-8 decoder at the end. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
484 if (u != NULL) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
485
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
486 /* Optimization for empty strings */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
487 if (size == 0 && unicode_empty != NULL) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
488 Py_INCREF(unicode_empty);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
489 return (PyObject *)unicode_empty;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
490 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
491
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
492 /* Single characters are shared when using this constructor.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
493 Restrict to ASCII, since the input must be UTF-8. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
494 if (size == 1 && Py_CHARMASK(*u) < 128) {
46149
7fdff02e6f45 Revert r61969 which added casts to Py_CHARMASK to avoid compiler warnings.
Neal Norwitz <nnorwitz@gmail.com>
parents: 46132
diff changeset
495 unicode = unicode_latin1[Py_CHARMASK(*u)];
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
496 if (!unicode) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
497 unicode = _PyUnicode_New(1);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
498 if (!unicode)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
499 return NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
500 unicode->str[0] = Py_CHARMASK(*u);
46149
7fdff02e6f45 Revert r61969 which added casts to Py_CHARMASK to avoid compiler warnings.
Neal Norwitz <nnorwitz@gmail.com>
parents: 46132
diff changeset
501 unicode_latin1[Py_CHARMASK(*u)] = unicode;
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
502 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
503 Py_INCREF(unicode);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
504 return (PyObject *)unicode;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
505 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
506
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
507 return PyUnicode_DecodeUTF8(u, size, NULL);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
508 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
509
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
510 unicode = _PyUnicode_New(size);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
511 if (!unicode)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
512 return NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
513
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
514 return (PyObject *)unicode;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
515 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
516
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
517 PyObject *PyUnicode_FromString(const char *u)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
518 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
519 size_t size = strlen(u);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
520 if (size > PY_SSIZE_T_MAX) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
521 PyErr_SetString(PyExc_OverflowError, "input too long");
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
522 return NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
523 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
524
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
525 return PyUnicode_FromStringAndSize(u, size);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
526 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
527
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
528 #ifdef HAVE_WCHAR_H
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
529
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
530 PyObject *PyUnicode_FromWideChar(register const wchar_t *w,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
531 Py_ssize_t size)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
532 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
533 PyUnicodeObject *unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
534
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
535 if (w == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
536 PyErr_BadInternalCall();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
537 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
538 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
539
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
540 unicode = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
541 if (!unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
542 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
543
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
544 /* Copy the wchar_t data into the new object */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
545 #ifdef HAVE_USABLE_WCHAR_T
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
546 memcpy(unicode->str, w, size * sizeof(wchar_t));
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
547 #else
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
548 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
549 register Py_UNICODE *u;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
550 register Py_ssize_t i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
551 u = PyUnicode_AS_UNICODE(unicode);
33766
1907ad54d351 Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar.
Marc-André Lemburg <mal@egenix.com>
parents: 33499
diff changeset
552 for (i = size; i > 0; i--)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
553 *u++ = *w++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
554 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
555 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
556
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
557 return (PyObject *)unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
558 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
559
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
560 static void
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
561 makefmt(char *fmt, int longflag, int size_tflag, int zeropad, int width, int precision, char c)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
562 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
563 *fmt++ = '%';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
564 if (width) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
565 if (zeropad)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
566 *fmt++ = '0';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
567 fmt += sprintf(fmt, "%d", width);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
568 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
569 if (precision)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
570 fmt += sprintf(fmt, ".%d", precision);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
571 if (longflag)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
572 *fmt++ = 'l';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
573 else if (size_tflag) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
574 char *f = PY_FORMAT_SIZE_T;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
575 while (*f)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
576 *fmt++ = *f++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
577 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
578 *fmt++ = c;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
579 *fmt = '\0';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
580 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
581
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
582 #define appendstring(string) {for (copy = string;*copy;) *s++ = *copy++;}
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
583
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
584 PyObject *
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
585 PyUnicode_FromFormatV(const char *format, va_list vargs)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
586 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
587 va_list count;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
588 Py_ssize_t callcount = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
589 PyObject **callresults = NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
590 PyObject **callresult = NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
591 Py_ssize_t n = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
592 int width = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
593 int precision = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
594 int zeropad;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
595 const char* f;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
596 Py_UNICODE *s;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
597 PyObject *string;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
598 /* used by sprintf */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
599 char buffer[21];
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
600 /* use abuffer instead of buffer, if we need more space
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
601 * (which can happen if there's a format specifier with width). */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
602 char *abuffer = NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
603 char *realbuffer;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
604 Py_ssize_t abuffersize = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
605 char fmt[60]; /* should be enough for %0width.precisionld */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
606 const char *copy;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
607
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
608 #ifdef VA_LIST_IS_ARRAY
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
609 Py_MEMCPY(count, vargs, sizeof(va_list));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
610 #else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
611 #ifdef __va_copy
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
612 __va_copy(count, vargs);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
613 #else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
614 count = vargs;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
615 #endif
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
616 #endif
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
617 /* step 1: count the number of %S/%R format specifications
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
618 * (we call PyObject_Str()/PyObject_Repr() for these objects
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
619 * once during step 3 and put the result in an array) */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
620 for (f = format; *f; f++) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
621 if (*f == '%' && (*(f+1)=='S' || *(f+1)=='R'))
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
622 ++callcount;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
623 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
624 /* step 2: allocate memory for the results of
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
625 * PyObject_Str()/PyObject_Repr() calls */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
626 if (callcount) {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
627 callresults = PyObject_Malloc(sizeof(PyObject *)*callcount);
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
628 if (!callresults) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
629 PyErr_NoMemory();
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
630 return NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
631 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
632 callresult = callresults;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
633 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
634 /* step 3: figure out how large a buffer we need */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
635 for (f = format; *f; f++) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
636 if (*f == '%') {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
637 const char* p = f;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
638 width = 0;
46026
f1ecb3e978db Remove compiler warnings (on Alpha at least) about using chars as
Neal Norwitz <nnorwitz@gmail.com>
parents: 45804
diff changeset
639 while (isdigit((unsigned)*f))
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
640 width = (width*10) + *f++ - '0';
46026
f1ecb3e978db Remove compiler warnings (on Alpha at least) about using chars as
Neal Norwitz <nnorwitz@gmail.com>
parents: 45804
diff changeset
641 while (*++f && *f != '%' && !isalpha((unsigned)*f))
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
642 ;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
643
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
644 /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
645 * they don't affect the amount of space we reserve.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
646 */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
647 if ((*f == 'l' || *f == 'z') &&
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
648 (f[1] == 'd' || f[1] == 'u'))
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
649 ++f;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
650
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
651 switch (*f) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
652 case 'c':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
653 (void)va_arg(count, int);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
654 /* fall through... */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
655 case '%':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
656 n++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
657 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
658 case 'd': case 'u': case 'i': case 'x':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
659 (void) va_arg(count, int);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
660 /* 20 bytes is enough to hold a 64-bit
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
661 integer. Decimal takes the most space.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
662 This isn't enough for octal.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
663 If a width is specified we need more
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
664 (which we allocate later). */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
665 if (width < 20)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
666 width = 20;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
667 n += width;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
668 if (abuffersize < width)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
669 abuffersize = width;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
670 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
671 case 's':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
672 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
673 /* UTF-8 */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
674 unsigned char*s;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
675 s = va_arg(count, unsigned char*);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
676 while (*s) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
677 if (*s < 128) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
678 n++; s++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
679 } else if (*s < 0xc0) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
680 /* invalid UTF-8 */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
681 n++; s++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
682 } else if (*s < 0xc0) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
683 n++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
684 s++; if(!*s)break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
685 s++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
686 } else if (*s < 0xe0) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
687 n++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
688 s++; if(!*s)break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
689 s++; if(!*s)break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
690 s++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
691 } else {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
692 #ifdef Py_UNICODE_WIDE
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
693 n++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
694 #else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
695 n+=2;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
696 #endif
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
697 s++; if(!*s)break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
698 s++; if(!*s)break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
699 s++; if(!*s)break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
700 s++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
701 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
702 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
703 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
704 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
705 case 'U':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
706 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
707 PyObject *obj = va_arg(count, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
708 assert(obj && PyUnicode_Check(obj));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
709 n += PyUnicode_GET_SIZE(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
710 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
711 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
712 case 'V':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
713 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
714 PyObject *obj = va_arg(count, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
715 const char *str = va_arg(count, const char *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
716 assert(obj || str);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
717 assert(!obj || PyUnicode_Check(obj));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
718 if (obj)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
719 n += PyUnicode_GET_SIZE(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
720 else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
721 n += strlen(str);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
722 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
723 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
724 case 'S':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
725 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
726 PyObject *obj = va_arg(count, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
727 PyObject *str;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
728 assert(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
729 str = PyObject_Str(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
730 if (!str)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
731 goto fail;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
732 n += PyUnicode_GET_SIZE(str);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
733 /* Remember the str and switch to the next slot */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
734 *callresult++ = str;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
735 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
736 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
737 case 'R':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
738 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
739 PyObject *obj = va_arg(count, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
740 PyObject *repr;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
741 assert(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
742 repr = PyObject_Repr(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
743 if (!repr)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
744 goto fail;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
745 n += PyUnicode_GET_SIZE(repr);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
746 /* Remember the repr and switch to the next slot */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
747 *callresult++ = repr;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
748 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
749 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
750 case 'p':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
751 (void) va_arg(count, int);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
752 /* maximum 64-bit pointer representation:
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
753 * 0xffffffffffffffff
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
754 * so 19 characters is enough.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
755 * XXX I count 18 -- what's the extra for?
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
756 */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
757 n += 19;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
758 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
759 default:
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
760 /* if we stumble upon an unknown
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
761 formatting code, copy the rest of
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
762 the format string to the output
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
763 string. (we cannot just skip the
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
764 code, since there's no way to know
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
765 what's in the argument list) */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
766 n += strlen(p);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
767 goto expand;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
768 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
769 } else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
770 n++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
771 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
772 expand:
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
773 if (abuffersize > 20) {
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
774 abuffer = PyObject_Malloc(abuffersize);
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
775 if (!abuffer) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
776 PyErr_NoMemory();
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
777 goto fail;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
778 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
779 realbuffer = abuffer;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
780 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
781 else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
782 realbuffer = buffer;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
783 /* step 4: fill the buffer */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
784 /* Since we've analyzed how much space we need for the worst case,
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
785 we don't have to resize the string.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
786 There can be no errors beyond this point. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
787 string = PyUnicode_FromUnicode(NULL, n);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
788 if (!string)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
789 goto fail;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
790
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
791 s = PyUnicode_AS_UNICODE(string);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
792 callresult = callresults;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
793
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
794 for (f = format; *f; f++) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
795 if (*f == '%') {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
796 const char* p = f++;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
797 int longflag = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
798 int size_tflag = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
799 zeropad = (*f == '0');
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
800 /* parse the width.precision part */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
801 width = 0;
46026
f1ecb3e978db Remove compiler warnings (on Alpha at least) about using chars as
Neal Norwitz <nnorwitz@gmail.com>
parents: 45804
diff changeset
802 while (isdigit((unsigned)*f))
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
803 width = (width*10) + *f++ - '0';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
804 precision = 0;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
805 if (*f == '.') {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
806 f++;
46026
f1ecb3e978db Remove compiler warnings (on Alpha at least) about using chars as
Neal Norwitz <nnorwitz@gmail.com>
parents: 45804
diff changeset
807 while (isdigit((unsigned)*f))
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
808 precision = (precision*10) + *f++ - '0';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
809 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
810 /* handle the long flag, but only for %ld and %lu.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
811 others can be added when necessary. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
812 if (*f == 'l' && (f[1] == 'd' || f[1] == 'u')) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
813 longflag = 1;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
814 ++f;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
815 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
816 /* handle the size_t flag. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
817 if (*f == 'z' && (f[1] == 'd' || f[1] == 'u')) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
818 size_tflag = 1;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
819 ++f;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
820 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
821
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
822 switch (*f) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
823 case 'c':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
824 *s++ = va_arg(vargs, int);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
825 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
826 case 'd':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
827 makefmt(fmt, longflag, size_tflag, zeropad, width, precision, 'd');
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
828 if (longflag)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
829 sprintf(realbuffer, fmt, va_arg(vargs, long));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
830 else if (size_tflag)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
831 sprintf(realbuffer, fmt, va_arg(vargs, Py_ssize_t));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
832 else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
833 sprintf(realbuffer, fmt, va_arg(vargs, int));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
834 appendstring(realbuffer);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
835 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
836 case 'u':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
837 makefmt(fmt, longflag, size_tflag, zeropad, width, precision, 'u');
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
838 if (longflag)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
839 sprintf(realbuffer, fmt, va_arg(vargs, unsigned long));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
840 else if (size_tflag)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
841 sprintf(realbuffer, fmt, va_arg(vargs, size_t));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
842 else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
843 sprintf(realbuffer, fmt, va_arg(vargs, unsigned int));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
844 appendstring(realbuffer);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
845 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
846 case 'i':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
847 makefmt(fmt, 0, 0, zeropad, width, precision, 'i');
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
848 sprintf(realbuffer, fmt, va_arg(vargs, int));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
849 appendstring(realbuffer);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
850 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
851 case 'x':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
852 makefmt(fmt, 0, 0, zeropad, width, precision, 'x');
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
853 sprintf(realbuffer, fmt, va_arg(vargs, int));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
854 appendstring(realbuffer);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
855 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
856 case 's':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
857 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
858 /* Parameter must be UTF-8 encoded.
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
859 In case of encoding errors, use
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
860 the replacement character. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
861 PyObject *u;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
862 p = va_arg(vargs, char*);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
863 u = PyUnicode_DecodeUTF8(p, strlen(p),
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
864 "replace");
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
865 if (!u)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
866 goto fail;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
867 Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(u),
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
868 PyUnicode_GET_SIZE(u));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
869 s += PyUnicode_GET_SIZE(u);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
870 Py_DECREF(u);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
871 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
872 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
873 case 'U':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
874 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
875 PyObject *obj = va_arg(vargs, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
876 Py_ssize_t size = PyUnicode_GET_SIZE(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
877 Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(obj), size);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
878 s += size;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
879 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
880 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
881 case 'V':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
882 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
883 PyObject *obj = va_arg(vargs, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
884 const char *str = va_arg(vargs, const char *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
885 if (obj) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
886 Py_ssize_t size = PyUnicode_GET_SIZE(obj);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
887 Py_UNICODE_COPY(s, PyUnicode_AS_UNICODE(obj), size);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
888 s += size;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
889 } else {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
890 appendstring(str);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
891 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
892 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
893 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
894 case 'S':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
895 case 'R':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
896 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
897 Py_UNICODE *ucopy;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
898 Py_ssize_t usize;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
899 Py_ssize_t upos;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
900 /* unused, since we already have the result */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
901 (void) va_arg(vargs, PyObject *);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
902 ucopy = PyUnicode_AS_UNICODE(*callresult);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
903 usize = PyUnicode_GET_SIZE(*callresult);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
904 for (upos = 0; upos<usize;)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
905 *s++ = ucopy[upos++];
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
906 /* We're done with the unicode()/repr() => forget it */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
907 Py_DECREF(*callresult);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
908 /* switch to next unicode()/repr() result */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
909 ++callresult;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
910 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
911 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
912 case 'p':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
913 sprintf(buffer, "%p", va_arg(vargs, void*));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
914 /* %p is ill-defined: ensure leading 0x. */
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
915 if (buffer[1] == 'X')
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
916 buffer[1] = 'x';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
917 else if (buffer[1] != 'x') {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
918 memmove(buffer+2, buffer, strlen(buffer)+1);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
919 buffer[0] = '0';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
920 buffer[1] = 'x';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
921 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
922 appendstring(buffer);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
923 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
924 case '%':
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
925 *s++ = '%';
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
926 break;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
927 default:
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
928 appendstring(p);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
929 goto end;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
930 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
931 } else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
932 *s++ = *f;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
933 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
934
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
935 end:
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
936 if (callresults)
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
937 PyObject_Free(callresults);
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
938 if (abuffer)
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
939 PyObject_Free(abuffer);
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
940 _PyUnicode_Resize(&string, s - PyUnicode_AS_UNICODE(string));
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
941 return string;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
942 fail:
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
943 if (callresults) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
944 PyObject **callresult2 = callresults;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
945 while (callresult2 < callresult) {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
946 Py_DECREF(*callresult2);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
947 ++callresult2;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
948 }
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
949 PyObject_Free(callresults);
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
950 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
951 if (abuffer)
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
952 PyObject_Free(abuffer);
44945
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
953 return NULL;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
954 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
955
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
956 #undef appendstring
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
957
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
958 PyObject *
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
959 PyUnicode_FromFormat(const char *format, ...)
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
960 {
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
961 PyObject* ret;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
962 va_list vargs;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
963
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
964 #ifdef HAVE_STDARG_PROTOTYPES
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
965 va_start(vargs, format);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
966 #else
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
967 va_start(vargs);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
968 #endif
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
969 ret = PyUnicode_FromFormatV(format, vargs);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
970 va_end(vargs);
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
971 return ret;
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
972 }
bbc719a8b58b Backport of several functions from Python 3.0 to 2.6 including PyUnicode_FromString, PyUnicode_Format and PyLong_From/AsSsize_t. The functions are partly required for the backport of the bytearray type and _fileio module. They should also make it easier to port C to 3.0.
Christian Heimes <christian@cheimes.de>
parents: 44439
diff changeset
973
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
974 Py_ssize_t PyUnicode_AsWideChar(PyUnicodeObject *unicode,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
975 wchar_t *w,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
976 Py_ssize_t size)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
977 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
978 if (unicode == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
979 PyErr_BadInternalCall();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
980 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
981 }
34016
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
982
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
983 /* If possible, try to copy the 0-termination as well */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
984 if (size > PyUnicode_GET_SIZE(unicode))
34016
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
985 size = PyUnicode_GET_SIZE(unicode) + 1;
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
986
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
987 #ifdef HAVE_USABLE_WCHAR_T
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
988 memcpy(w, unicode->str, size * sizeof(wchar_t));
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
989 #else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
990 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
991 register Py_UNICODE *u;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
992 register Py_ssize_t i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
993 u = PyUnicode_AS_UNICODE(unicode);
33766
1907ad54d351 Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar.
Marc-André Lemburg <mal@egenix.com>
parents: 33499
diff changeset
994 for (i = size; i > 0; i--)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
995 *w++ = *u++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
996 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
997 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
998
34016
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
999 if (size > PyUnicode_GET_SIZE(unicode))
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
1000 return PyUnicode_GET_SIZE(unicode);
264e45baacd5 Correct the handling of 0-termination of PyUnicode_AsWideChar()
Marc-André Lemburg <mal@egenix.com>
parents: 33766
diff changeset
1001 else
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1002 return size;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1003 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1004
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1005 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1006
24737
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1007 PyObject *PyUnicode_FromOrdinal(int ordinal)
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1008 {
31873
227cb7402c6e SF Patch #926375: Remove a useless UTF-16 support code that is never
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31455
diff changeset
1009 Py_UNICODE s[1];
24737
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1010
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1011 #ifdef Py_UNICODE_WIDE
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1012 if (ordinal < 0 || ordinal > 0x10ffff) {
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1013 PyErr_SetString(PyExc_ValueError,
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1014 "unichr() arg not in range(0x110000) "
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1015 "(wide Python build)");
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1016 return NULL;
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1017 }
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1018 #else
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1019 if (ordinal < 0 || ordinal > 0xffff) {
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1020 PyErr_SetString(PyExc_ValueError,
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1021 "unichr() arg not in range(0x10000) "
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1022 "(narrow Python build)");
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1023 return NULL;
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1024 }
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1025 #endif
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1026
31873
227cb7402c6e SF Patch #926375: Remove a useless UTF-16 support code that is never
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31455
diff changeset
1027 s[0] = (Py_UNICODE)ordinal;
227cb7402c6e SF Patch #926375: Remove a useless UTF-16 support code that is never
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31455
diff changeset
1028 return PyUnicode_FromUnicode(s, 1);
24737
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1029 }
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
1030
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1031 PyObject *PyUnicode_FromObject(register PyObject *obj)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1032 {
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1033 /* XXX Perhaps we should make this API an alias of
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1034 PyObject_Unicode() instead ?! */
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1035 if (PyUnicode_CheckExact(obj)) {
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1036 Py_INCREF(obj);
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1037 return obj;
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1038 }
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1039 if (PyUnicode_Check(obj)) {
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1040 /* For a Unicode subtype that's not a Unicode object,
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1041 return a true Unicode object with the same data. */
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1042 return PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(obj),
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1043 PyUnicode_GET_SIZE(obj));
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1044 }
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1045 return PyUnicode_FromEncodedObject(obj, NULL, "strict");
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1046 }
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1047
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1048 PyObject *PyUnicode_FromEncodedObject(register PyObject *obj,
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1049 const char *encoding,
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1050 const char *errors)
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1051 {
19999
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1052 const char *s = NULL;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1053 Py_ssize_t len;
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1054 PyObject *v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1055
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1056 if (obj == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1057 PyErr_BadInternalCall();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1058 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1059 }
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1060
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1061 #if 0
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1062 /* For b/w compatibility we also accept Unicode objects provided
20563
9058e49c6c57 Additional test and documentation for the unicode() changes.
Marc-André Lemburg <mal@egenix.com>
parents: 20554
diff changeset
1063 that no encodings is given and then redirect to
9058e49c6c57 Additional test and documentation for the unicode() changes.
Marc-André Lemburg <mal@egenix.com>
parents: 20554
diff changeset
1064 PyObject_Unicode() which then applies the additional logic for
9058e49c6c57 Additional test and documentation for the unicode() changes.
Marc-André Lemburg <mal@egenix.com>
parents: 20554
diff changeset
1065 Unicode subclasses.
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1066
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1067 NOTE: This API should really only be used for object which
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1068 represent *encoded* Unicode !
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1069
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1070 */
19999
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1071 if (PyUnicode_Check(obj)) {
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1072 if (encoding) {
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1073 PyErr_SetString(PyExc_TypeError,
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1074 "decoding Unicode is not supported");
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1075 return NULL;
19999
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1076 }
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1077 return PyObject_Unicode(obj);
19999
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1078 }
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1079 #else
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1080 if (PyUnicode_Check(obj)) {
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1081 PyErr_SetString(PyExc_TypeError,
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1082 "decoding Unicode is not supported");
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1083 return NULL;
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1084 }
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1085 #endif
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1086
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1087 /* Coerce object */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1088 if (PyString_Check(obj)) {
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1089 s = PyString_AS_STRING(obj);
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1090 len = PyString_GET_SIZE(obj);
46107
5341b30b1812 Merged revisions 61750,61752,61754,61756,61760,61763,61768,61772,61775,61805,61809,61812,61819,61917,61920,61930,61933-61934 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46062
diff changeset
1091 }
47291
04bd3690fb1f First step of the C API rename:
Christian Heimes <christian@cheimes.de>
parents: 46908
diff changeset
1092 else if (PyByteArray_Check(obj)) {
46107
5341b30b1812 Merged revisions 61750,61752,61754,61756,61760,61763,61768,61772,61775,61805,61809,61812,61819,61917,61920,61930,61933-61934 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46062
diff changeset
1093 /* Python 2.x specific */
5341b30b1812 Merged revisions 61750,61752,61754,61756,61760,61763,61768,61772,61775,61805,61809,61812,61819,61917,61920,61930,61933-61934 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46062
diff changeset
1094 PyErr_Format(PyExc_TypeError,
5341b30b1812 Merged revisions 61750,61752,61754,61756,61760,61763,61768,61772,61775,61805,61809,61812,61819,61917,61920,61930,61933-61934 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46062
diff changeset
1095 "decoding bytearray is not supported");
5341b30b1812 Merged revisions 61750,61752,61754,61756,61760,61763,61768,61772,61775,61805,61809,61812,61819,61917,61920,61930,61933-61934 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46062
diff changeset
1096 return NULL;
5341b30b1812 Merged revisions 61750,61752,61754,61756,61760,61763,61768,61772,61775,61805,61809,61812,61819,61917,61920,61930,61933-61934 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46062
diff changeset
1097 }
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1098 else if (PyObject_AsCharBuffer(obj, &s, &len)) {
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1099 /* Overwrite the error message with something more useful in
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1100 case of a TypeError. */
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1101 if (PyErr_ExceptionMatches(PyExc_TypeError))
19999
1f4e2627b627 Implement the changes proposed in patch #413333. unicode(obj) now
Marc-André Lemburg <mal@egenix.com>
parents: 19998
diff changeset
1102 PyErr_Format(PyExc_TypeError,
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1103 "coercing to Unicode: need string or buffer, "
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
1104 "%.80s found",
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
1105 Py_TYPE(obj)->tp_name);
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1106 goto onError;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
1107 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1108
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1109 /* Convert to Unicode */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1110 if (len == 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1111 Py_INCREF(unicode_empty);
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1112 v = (PyObject *)unicode_empty;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1113 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1114 else
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1115 v = PyUnicode_Decode(s, len, encoding, errors);
15952
1f92d63b20f0 This patch adds a new builtin unistr() which behaves like str()
Marc-André Lemburg <mal@egenix.com>
parents: 15910
diff changeset
1116
13401
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1117 return v;
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1118
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1119 onError:
8f1a2deeb9b0 Added new API PyUnicode_FromEncodedObject() which supports decoding
Marc-André Lemburg <mal@egenix.com>
parents: 13396
diff changeset
1120 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1121 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1122
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1123 PyObject *PyUnicode_Decode(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1124 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1125 const char *encoding,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1126 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1127 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1128 PyObject *buffer = NULL, *unicode;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1129
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1130 if (encoding == NULL)
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1131 encoding = PyUnicode_GetDefaultEncoding();
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1132
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1133 /* Shortcuts for common default encodings */
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1134 if (strcmp(encoding, "utf-8") == 0)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1135 return PyUnicode_DecodeUTF8(s, size, errors);
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1136 else if (strcmp(encoding, "latin-1") == 0)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1137 return PyUnicode_DecodeLatin1(s, size, errors);
29706
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1138 #if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T)
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1139 else if (strcmp(encoding, "mbcs") == 0)
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1140 return PyUnicode_DecodeMBCS(s, size, errors);
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1141 #endif
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1142 else if (strcmp(encoding, "ascii") == 0)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1143 return PyUnicode_DecodeASCII(s, size, errors);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1144
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1145 /* Decode via the codec registry */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1146 buffer = PyBuffer_FromMemory((void *)s, size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1147 if (buffer == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1148 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1149 unicode = PyCodec_Decode(buffer, encoding, errors);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1150 if (unicode == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1151 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1152 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1153 PyErr_Format(PyExc_TypeError,
12320
5de0b939fe61 Skip Montanaro: add string precisions to calls to PyErr_Format
Guido van Rossum <guido@python.org>
parents: 12275
diff changeset
1154 "decoder did not return an unicode object (type=%.400s)",
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
1155 Py_TYPE(unicode)->tp_name);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1156 Py_DECREF(unicode);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1157 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1158 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1159 Py_DECREF(buffer);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1160 return unicode;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1161
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1162 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1163 Py_XDECREF(buffer);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1164 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1165 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1166
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1167 PyObject *PyUnicode_AsDecodedObject(PyObject *unicode,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1168 const char *encoding,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1169 const char *errors)
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1170 {
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1171 PyObject *v;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1172
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1173 if (!PyUnicode_Check(unicode)) {
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1174 PyErr_BadArgument();
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1175 goto onError;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1176 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1177
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1178 if (encoding == NULL)
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1179 encoding = PyUnicode_GetDefaultEncoding();
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1180
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1181 /* Decode via the codec registry */
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1182 v = PyCodec_Decode(unicode, encoding, errors);
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1183 if (v == NULL)
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1184 goto onError;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1185 return v;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1186
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1187 onError:
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1188 return NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1189 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1190
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1191 PyObject *PyUnicode_Encode(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1192 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1193 const char *encoding,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1194 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1195 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1196 PyObject *v, *unicode;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1197
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1198 unicode = PyUnicode_FromUnicode(s, size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1199 if (unicode == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1200 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1201 v = PyUnicode_AsEncodedString(unicode, encoding, errors);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1202 Py_DECREF(unicode);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1203 return v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1204 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1205
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1206 PyObject *PyUnicode_AsEncodedObject(PyObject *unicode,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1207 const char *encoding,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1208 const char *errors)
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1209 {
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1210 PyObject *v;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1211
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1212 if (!PyUnicode_Check(unicode)) {
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1213 PyErr_BadArgument();
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1214 goto onError;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1215 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1216
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1217 if (encoding == NULL)
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1218 encoding = PyUnicode_GetDefaultEncoding();
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1219
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1220 /* Encode via the codec registry */
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1221 v = PyCodec_Encode(unicode, encoding, errors);
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1222 if (v == NULL)
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1223 goto onError;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1224 return v;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1225
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1226 onError:
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1227 return NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1228 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
1229
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1230 PyObject *PyUnicode_AsEncodedString(PyObject *unicode,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1231 const char *encoding,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1232 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1233 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1234 PyObject *v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1235
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1236 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1237 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1238 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1239 }
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1240
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1241 if (encoding == NULL)
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1242 encoding = PyUnicode_GetDefaultEncoding();
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1243
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1244 /* Shortcuts for common default encodings */
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1245 if (errors == NULL) {
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1246 if (strcmp(encoding, "utf-8") == 0)
17995
837825a779c1 fix bogus indentation
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 17887
diff changeset
1247 return PyUnicode_AsUTF8String(unicode);
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1248 else if (strcmp(encoding, "latin-1") == 0)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1249 return PyUnicode_AsLatin1String(unicode);
29706
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1250 #if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T)
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1251 else if (strcmp(encoding, "mbcs") == 0)
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1252 return PyUnicode_AsMBCSString(unicode);
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
1253 #endif
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1254 else if (strcmp(encoding, "ascii") == 0)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1255 return PyUnicode_AsASCIIString(unicode);
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1256 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1257
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1258 /* Encode via the codec registry */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1259 v = PyCodec_Encode(unicode, encoding, errors);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1260 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1261 goto onError;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1262 if (!PyString_Check(v)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1263 PyErr_Format(PyExc_TypeError,
12320
5de0b939fe61 Skip Montanaro: add string precisions to calls to PyErr_Format
Guido van Rossum <guido@python.org>
parents: 12275
diff changeset
1264 "encoder did not return a string object (type=%.400s)",
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
1265 Py_TYPE(v)->tp_name);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1266 Py_DECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1267 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1268 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1269 return v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1270
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1271 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1272 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1273 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1274
13837
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1275 PyObject *_PyUnicode_AsDefaultEncodedString(PyObject *unicode,
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1276 const char *errors)
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1277 {
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1278 PyObject *v = ((PyUnicodeObject *)unicode)->defenc;
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1279
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1280 if (v)
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1281 return v;
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1282 v = PyUnicode_AsEncodedString(unicode, NULL, errors);
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1283 if (v && errors == NULL)
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1284 ((PyUnicodeObject *)unicode)->defenc = v;
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1285 return v;
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1286 }
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
1287
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1288 Py_UNICODE *PyUnicode_AsUnicode(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1289 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1290 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1291 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1292 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1293 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1294 return PyUnicode_AS_UNICODE(unicode);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1295
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1296 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1297 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1298 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1299
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1300 Py_ssize_t PyUnicode_GetSize(PyObject *unicode)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1301 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1302 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1303 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1304 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1305 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1306 return PyUnicode_GET_SIZE(unicode);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1307
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1308 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1309 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1310 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1311
13660
a1ce07218781 Miscelaneous ANSIfications. I'm assuming here 'main' should take (int,
Thomas Wouters <thomas@python.org>
parents: 13635
diff changeset
1312 const char *PyUnicode_GetDefaultEncoding(void)
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1313 {
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1314 return unicode_default_encoding;
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1315 }
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1316
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1317 int PyUnicode_SetDefaultEncoding(const char *encoding)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1318 {
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1319 PyObject *v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1320
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1321 /* Make sure the encoding is valid. As side effect, this also
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1322 loads the encoding into the codec registry cache. */
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1323 v = _PyCodec_Lookup(encoding);
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1324 if (v == NULL)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1325 goto onError;
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1326 Py_DECREF(v);
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1327 strncpy(unicode_default_encoding,
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1328 encoding,
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1329 sizeof(unicode_default_encoding));
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1330 return 0;
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1331
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1332 onError:
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1333 return -1;
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1334 }
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
1335
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1336 /* error handling callback helper:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1337 build arguments, call the callback and check the arguments,
35806
44fe157255bc fix typos, mostly in comments
Fred Drake <fdrake@acm.org>
parents: 35758
diff changeset
1338 if no exception occurred, copy the replacement to the output
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1339 and adjust various state variables.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1340 return 0 on success, -1 on error
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1341 */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1342
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1343 static
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1344 int unicode_decode_call_errorhandler(const char *errors, PyObject **errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1345 const char *encoding, const char *reason,
43052
973da2ebc2d0 Rewrap line.
Walter Dörwald <walter@livinglogic.de>
parents: 42951
diff changeset
1346 const char *input, Py_ssize_t insize, Py_ssize_t *startinpos,
973da2ebc2d0 Rewrap line.
Walter Dörwald <walter@livinglogic.de>
parents: 42951
diff changeset
1347 Py_ssize_t *endinpos, PyObject **exceptionObject, const char **inptr,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1348 PyObject **output, Py_ssize_t *outpos, Py_UNICODE **outptr)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1349 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1350 static char *argparse = "O!n;decoding error handler must return (unicode, int) tuple";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1351
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1352 PyObject *restuple = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1353 PyObject *repunicode = NULL;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1354 Py_ssize_t outsize = PyUnicode_GET_SIZE(*output);
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1355 Py_ssize_t requiredsize;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1356 Py_ssize_t newpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1357 Py_UNICODE *repptr;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1358 Py_ssize_t repsize;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1359 int res = -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1360
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1361 if (*errorHandler == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1362 *errorHandler = PyCodec_LookupError(errors);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1363 if (*errorHandler == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1364 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1365 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1366
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1367 if (*exceptionObject == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1368 *exceptionObject = PyUnicodeDecodeError_Create(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1369 encoding, input, insize, *startinpos, *endinpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1370 if (*exceptionObject == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1371 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1372 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1373 else {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1374 if (PyUnicodeDecodeError_SetStart(*exceptionObject, *startinpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1375 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1376 if (PyUnicodeDecodeError_SetEnd(*exceptionObject, *endinpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1377 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1378 if (PyUnicodeDecodeError_SetReason(*exceptionObject, reason))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1379 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1380 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1381
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1382 restuple = PyObject_CallFunctionObjArgs(*errorHandler, *exceptionObject, NULL);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1383 if (restuple == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1384 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1385 if (!PyTuple_Check(restuple)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1386 PyErr_Format(PyExc_TypeError, &argparse[4]);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1387 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1388 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1389 if (!PyArg_ParseTuple(restuple, argparse, &PyUnicode_Type, &repunicode, &newpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1390 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1391 if (newpos<0)
27555
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
1392 newpos = insize+newpos;
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
1393 if (newpos<0 || newpos>insize) {
36510
0f3894bb2ada Support %zd in PyErr_Format and PyString_FromFormat.
Martin v. Löwis <martin@v.loewis.de>
parents: 36508
diff changeset
1394 PyErr_Format(PyExc_IndexError, "position %zd from error handler out of bounds", newpos);
27555
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
1395 goto onError;
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
1396 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1397
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1398 /* need more space? (at least enough for what we
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1399 have+the replacement+the rest of the string (starting
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1400 at the new input position), so we won't have to check space
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1401 when there are no errors in the rest of the string) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1402 repptr = PyUnicode_AS_UNICODE(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1403 repsize = PyUnicode_GET_SIZE(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1404 requiredsize = *outpos + repsize + insize-newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1405 if (requiredsize > outsize) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1406 if (requiredsize<2*outsize)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1407 requiredsize = 2*outsize;
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
1408 if (PyUnicode_Resize(output, requiredsize) < 0)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1409 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1410 *outptr = PyUnicode_AS_UNICODE(*output) + *outpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1411 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1412 *endinpos = newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1413 *inptr = input + newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1414 Py_UNICODE_COPY(*outptr, repptr, repsize);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1415 *outptr += repsize;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1416 *outpos += repsize;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1417 /* we made it! */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1418 res = 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1419
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1420 onError:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1421 Py_XDECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1422 return res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1423 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1424
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1425 /* --- UTF-7 Codec -------------------------------------------------------- */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1426
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1427 /* see RFC2152 for details */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1428
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1429 static
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1430 char utf7_special[128] = {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1431 /* indicate whether a UTF-7 character is special i.e. cannot be directly
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1432 encoded:
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1433 0 - not special
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1434 1 - special
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1435 2 - whitespace (optional)
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1436 3 - RFC2152 Set O (optional) */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1437 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1, 2, 1, 1,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1438 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1439 2, 3, 3, 3, 3, 3, 3, 0, 0, 0, 3, 1, 0, 0, 0, 1,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1440 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3, 0,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1441 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1442 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 1, 3, 3, 3,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1443 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1444 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 1, 1,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1445
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1446 };
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1447
35728
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1448 /* Note: The comparison (c) <= 0 is a trick to work-around gcc
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1449 warnings about the comparison always being false; since
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1450 utf7_special[0] is 1, we can safely make that one comparison
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1451 true */
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1452
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1453 #define SPECIAL(c, encodeO, encodeWS) \
35728
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1454 ((c) > 127 || (c) <= 0 || utf7_special[(c)] == 1 || \
35729
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1455 (encodeWS && (utf7_special[(c)] == 2)) || \
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1456 (encodeO && (utf7_special[(c)] == 3)))
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1457
35728
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1458 #define B64(n) \
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1459 ("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"[(n) & 0x3f])
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1460 #define B64CHAR(c) \
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1461 (isalnum(c) || (c) == '+' || (c) == '/')
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1462 #define UB64(c) \
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1463 ((c) == '+' ? 62 : (c) == '/' ? 63 : (c) >= 'a' ? \
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1464 (c) - 71 : (c) >= 'A' ? (c) - 65 : (c) + 4 )
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1465
35729
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1466 #define ENCODE(out, ch, bits) \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1467 while (bits >= 6) { \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1468 *out++ = B64(ch >> (bits-6)); \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1469 bits -= 6; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1470 }
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1471
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1472 #define DECODE(out, ch, bits, surrogate) \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1473 while (bits >= 16) { \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1474 Py_UNICODE outCh = (Py_UNICODE) ((ch >> (bits-16)) & 0xffff); \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1475 bits -= 16; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1476 if (surrogate) { \
35728
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1477 /* We have already generated an error for the high surrogate \
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1478 so let's not bother seeing if the low surrogate is correct or not */ \
35729
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1479 surrogate = 0; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1480 } else if (0xDC00 <= outCh && outCh <= 0xDFFF) { \
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1481 /* This is a surrogate pair. Unfortunately we can't represent \
35729
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1482 it in a 16-bit character */ \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1483 surrogate = 1; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1484 errmsg = "code pairs are not supported"; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1485 goto utf7Error; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1486 } else { \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1487 *out++ = outCh; \
6913abb7047a Whitespace corrections.
Marc-André Lemburg <mal@egenix.com>
parents: 35728
diff changeset
1488 } \
35728
9ed43e637ae7 Bug fix for [ 1331062 ] utf 7 codec broken.
Marc-André Lemburg <mal@egenix.com>
parents: 35702
diff changeset
1489 }
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1490
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1491 PyObject *PyUnicode_DecodeUTF7(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1492 Py_ssize_t size,
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1493 const char *errors)
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1494 {
43932
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1495 return PyUnicode_DecodeUTF7Stateful(s, size, errors, NULL);
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1496 }
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1497
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1498 PyObject *PyUnicode_DecodeUTF7Stateful(const char *s,
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1499 Py_ssize_t size,
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1500 const char *errors,
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1501 Py_ssize_t *consumed)
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1502 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1503 const char *starts = s;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1504 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1505 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1506 Py_ssize_t outpos;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1507 const char *e;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1508 PyUnicodeObject *unicode;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1509 Py_UNICODE *p;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1510 const char *errmsg = "";
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1511 int inShift = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1512 unsigned int bitsleft = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1513 unsigned long charsleft = 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1514 int surrogate = 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1515 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1516 PyObject *exc = NULL;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1517
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1518 unicode = _PyUnicode_New(size);
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1519 if (!unicode)
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1520 return NULL;
43932
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1521 if (size == 0) {
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1522 if (consumed)
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1523 *consumed = 0;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1524 return (PyObject *)unicode;
43932
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1525 }
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1526
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1527 p = unicode->str;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1528 e = s + size;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1529
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1530 while (s < e) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1531 Py_UNICODE ch;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1532 restart:
48070
35a462ff9873 #2242: utf7 decoding crashes on bogus input on some Windows/MSVC versions
Antoine Pitrou <solipsis@pitrou.net>
parents: 48003
diff changeset
1533 ch = (unsigned char) *s;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1534
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1535 if (inShift) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1536 if ((ch == '-') || !B64CHAR(ch)) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1537 inShift = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1538 s++;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1539
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1540 /* p, charsleft, bitsleft, surrogate = */ DECODE(p, charsleft, bitsleft, surrogate);
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1541 if (bitsleft >= 6) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1542 /* The shift sequence has a partial character in it. If
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1543 bitsleft < 6 then we could just classify it as padding
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1544 but that is not the case here */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1545
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1546 errmsg = "partial character in shift sequence";
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1547 goto utf7Error;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1548 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1549 /* According to RFC2152 the remaining bits should be zero. We
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1550 choose to signal an error/insert a replacement character
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1551 here so indicate the potential of a misencoded character. */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1552
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1553 /* On x86, a << b == a << (b%32) so make sure that bitsleft != 0 */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1554 if (bitsleft && charsleft << (sizeof(charsleft) * 8 - bitsleft)) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1555 errmsg = "non-zero padding bits in shift sequence";
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1556 goto utf7Error;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1557 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1558
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1559 if (ch == '-') {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1560 if ((s < e) && (*(s) == '-')) {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1561 *p++ = '-';
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1562 inShift = 1;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1563 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1564 } else if (SPECIAL(ch,0,0)) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1565 errmsg = "unexpected special character";
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1566 goto utf7Error;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1567 } else {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1568 *p++ = ch;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1569 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1570 } else {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1571 charsleft = (charsleft << 6) | UB64(ch);
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1572 bitsleft += 6;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1573 s++;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1574 /* p, charsleft, bitsleft, surrogate = */ DECODE(p, charsleft, bitsleft, surrogate);
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1575 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1576 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1577 else if ( ch == '+' ) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1578 startinpos = s-starts;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1579 s++;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1580 if (s < e && *s == '-') {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1581 s++;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1582 *p++ = '+';
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1583 } else
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1584 {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1585 inShift = 1;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1586 bitsleft = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1587 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1588 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1589 else if (SPECIAL(ch,0,0)) {
43053
6fcb873f396c Set startinpos before calling the error handler.
Walter Dörwald <walter@livinglogic.de>
parents: 43052
diff changeset
1590 startinpos = s-starts;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1591 errmsg = "unexpected special character";
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1592 s++;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1593 goto utf7Error;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1594 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1595 else {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1596 *p++ = ch;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1597 s++;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1598 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1599 continue;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1600 utf7Error:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1601 outpos = p-PyUnicode_AS_UNICODE(unicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1602 endinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1603 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1604 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1605 "utf7", errmsg,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1606 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1607 (PyObject **)&unicode, &outpos, &p))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1608 goto onError;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1609 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1610
43932
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1611 if (inShift && !consumed) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1612 outpos = p-PyUnicode_AS_UNICODE(unicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1613 endinpos = size;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1614 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1615 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1616 "utf7", "unterminated shift sequence",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1617 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1618 (PyObject **)&unicode, &outpos, &p))
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1619 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1620 if (s < e)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1621 goto restart;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1622 }
43932
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1623 if (consumed) {
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1624 if(inShift)
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1625 *consumed = startinpos;
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1626 else
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1627 *consumed = s-starts;
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43888
diff changeset
1628 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1629
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
1630 if (_PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1631 goto onError;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1632
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1633 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1634 Py_XDECREF(exc);
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1635 return (PyObject *)unicode;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1636
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1637 onError:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1638 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1639 Py_XDECREF(exc);
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1640 Py_DECREF(unicode);
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1641 return NULL;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1642 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1643
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1644
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1645 PyObject *PyUnicode_EncodeUTF7(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1646 Py_ssize_t size,
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1647 int encodeSetO,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1648 int encodeWhiteSpace,
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1649 const char *errors)
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1650 {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1651 PyObject *v;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1652 /* It might be possible to tighten this worst case */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1653 Py_ssize_t cbAllocated = 5 * size;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1654 int inShift = 0;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1655 Py_ssize_t i = 0;
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1656 unsigned int bitsleft = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1657 unsigned long charsleft = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1658 char * out;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1659 char * start;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1660
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
1661 if (cbAllocated / 5 != size)
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
1662 return PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
1663
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1664 if (size == 0)
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1665 return PyString_FromStringAndSize(NULL, 0);
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1666
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1667 v = PyString_FromStringAndSize(NULL, cbAllocated);
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1668 if (v == NULL)
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1669 return NULL;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1670
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1671 start = out = PyString_AS_STRING(v);
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1672 for (;i < size; ++i) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1673 Py_UNICODE ch = s[i];
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1674
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1675 if (!inShift) {
31325
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1676 if (ch == '+') {
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1677 *out++ = '+';
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1678 *out++ = '-';
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1679 } else if (SPECIAL(ch, encodeSetO, encodeWhiteSpace)) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1680 charsleft = ch;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1681 bitsleft = 16;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1682 *out++ = '+';
31325
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1683 /* out, charsleft, bitsleft = */ ENCODE(out, charsleft, bitsleft);
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1684 inShift = bitsleft > 0;
31325
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1685 } else {
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1686 *out++ = (char) ch;
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1687 }
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1688 } else {
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1689 if (!SPECIAL(ch, encodeSetO, encodeWhiteSpace)) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1690 *out++ = B64(charsleft << (6-bitsleft));
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1691 charsleft = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1692 bitsleft = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1693 /* Characters not in the BASE64 set implicitly unshift the sequence
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1694 so no '-' is required, except if the character is itself a '-' */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1695 if (B64CHAR(ch) || ch == '-') {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1696 *out++ = '-';
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1697 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1698 inShift = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1699 *out++ = (char) ch;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1700 } else {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1701 bitsleft += 16;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1702 charsleft = (charsleft << 16) | ch;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1703 /* out, charsleft, bitsleft = */ ENCODE(out, charsleft, bitsleft);
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1704
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1705 /* If the next character is special then we dont' need to terminate
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1706 the shift sequence. If the next character is not a BASE64 character
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1707 or '-' then the shift sequence will be terminated implicitly and we
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1708 don't have to insert a '-'. */
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1709
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1710 if (bitsleft == 0) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1711 if (i + 1 < size) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1712 Py_UNICODE ch2 = s[i+1];
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1713
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1714 if (SPECIAL(ch2, encodeSetO, encodeWhiteSpace)) {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1715
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1716 } else if (B64CHAR(ch2) || ch2 == '-') {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1717 *out++ = '-';
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1718 inShift = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1719 } else {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1720 inShift = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1721 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1722
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1723 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1724 else {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1725 *out++ = '-';
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1726 inShift = 0;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1727 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1728 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1729 }
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1730 }
31325
b4532c246a3a Cosmetic fix for wrongly indented tabs with ts=4.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31252
diff changeset
1731 }
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1732 if (bitsleft) {
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1733 *out++= B64(charsleft << (6-bitsleft) );
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1734 *out++ = '-';
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1735 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1736
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
1737 _PyString_Resize(&v, out - start);
19998
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1738 return v;
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1739 }
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1740
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1741 #undef SPECIAL
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1742 #undef B64
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1743 #undef B64CHAR
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1744 #undef UB64
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1745 #undef ENCODE
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1746 #undef DECODE
8a0c97453df4 Patch #435971: UTF-7 codec by Brian Quinlan.
Marc-André Lemburg <mal@egenix.com>
parents: 19886
diff changeset
1747
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1748 /* --- UTF-8 Codec -------------------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1749
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1750 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1751 char utf8_code_length[256] = {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1752 /* Map UTF-8 encoded prefix byte to sequence length. zero means
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1753 illegal prefix. see RFC 2279 for details */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1754 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1755 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1756 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1757 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1758 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1759 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1760 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1761 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1762 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1763 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1764 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1765 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1766 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1767 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1768 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1769 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 0, 0
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1770 };
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1771
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1772 PyObject *PyUnicode_DecodeUTF8(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1773 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1774 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1775 {
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1776 return PyUnicode_DecodeUTF8Stateful(s, size, errors, NULL);
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1777 }
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1778
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1779 PyObject *PyUnicode_DecodeUTF8Stateful(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1780 Py_ssize_t size,
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1781 const char *errors,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1782 Py_ssize_t *consumed)
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1783 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1784 const char *starts = s;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1785 int n;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1786 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1787 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1788 Py_ssize_t outpos;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1789 const char *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1790 PyUnicodeObject *unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1791 Py_UNICODE *p;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1792 const char *errmsg = "";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1793 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1794 PyObject *exc = NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1795
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1796 /* Note: size will always be longer than the resulting Unicode
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1797 character count */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1798 unicode = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1799 if (!unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1800 return NULL;
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1801 if (size == 0) {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1802 if (consumed)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1803 *consumed = 0;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1804 return (PyObject *)unicode;
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1805 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1806
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1807 /* Unpack UTF-8 encoded data */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1808 p = unicode->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1809 e = s + size;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1810
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1811 while (s < e) {
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1812 Py_UCS4 ch = (unsigned char)*s;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1813
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1814 if (ch < 0x80) {
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1815 *p++ = (Py_UNICODE)ch;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1816 s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1817 continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1818 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1819
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1820 n = utf8_code_length[ch];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1821
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1822 if (s + n > e) {
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1823 if (consumed)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1824 break;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1825 else {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1826 errmsg = "unexpected end of data";
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1827 startinpos = s-starts;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1828 endinpos = size;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1829 goto utf8Error;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1830 }
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1831 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1832
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1833 switch (n) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1834
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1835 case 0:
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1836 errmsg = "unexpected code byte";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1837 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1838 endinpos = startinpos+1;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1839 goto utf8Error;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1840
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1841 case 1:
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1842 errmsg = "internal error";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1843 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1844 endinpos = startinpos+1;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1845 goto utf8Error;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1846
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1847 case 2:
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1848 if ((s[1] & 0xc0) != 0x80) {
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1849 errmsg = "invalid data";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1850 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1851 endinpos = startinpos+2;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1852 goto utf8Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1853 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1854 ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f);
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1855 if (ch < 0x80) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1856 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1857 endinpos = startinpos+2;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1858 errmsg = "illegal encoding";
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1859 goto utf8Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1860 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1861 else
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1862 *p++ = (Py_UNICODE)ch;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1863 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1864
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1865 case 3:
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1866 if ((s[1] & 0xc0) != 0x80 ||
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1867 (s[2] & 0xc0) != 0x80) {
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1868 errmsg = "invalid data";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1869 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1870 endinpos = startinpos+3;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1871 goto utf8Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1872 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1873 ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f);
21886
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1874 if (ch < 0x0800) {
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1875 /* Note: UTF-8 encodings of surrogates are considered
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1876 legal UTF-8 sequences;
21886
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1877
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1878 XXX For wide builds (UCS-4) we should probably try
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1879 to recombine the surrogates into a single code
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1880 unit.
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1881 */
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1882 errmsg = "illegal encoding";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1883 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1884 endinpos = startinpos+3;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1885 goto utf8Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1886 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1887 else
21886
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1888 *p++ = (Py_UNICODE)ch;
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1889 break;
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1890
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1891 case 4:
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1892 if ((s[1] & 0xc0) != 0x80 ||
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1893 (s[2] & 0xc0) != 0x80 ||
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1894 (s[3] & 0xc0) != 0x80) {
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1895 errmsg = "invalid data";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1896 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1897 endinpos = startinpos+4;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1898 goto utf8Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1899 }
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1900 ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) +
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1901 ((s[2] & 0x3f) << 6) + (s[3] & 0x3f);
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1902 /* validate and convert to UTF-16 */
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
1903 if ((ch < 0x10000) /* minimum value allowed for 4
21886
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1904 byte encoding */
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
1905 || (ch > 0x10ffff)) /* maximum value allowed for
21886
abedb2418f62 Fix to the UTF-8 encoder: it failed on 0-length input strings.
Marc-André Lemburg <mal@egenix.com>
parents: 21884
diff changeset
1906 UTF-16 */
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
1907 {
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1908 errmsg = "illegal encoding";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1909 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1910 endinpos = startinpos+4;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1911 goto utf8Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1912 }
18324
3ba7a358ce02 use Py_UNICODE_WIDE instead of USE_UCS4_STORAGE and Py_UNICODE_SIZE
Fredrik Lundh <fredrik@pythonware.com>
parents: 18298
diff changeset
1913 #ifdef Py_UNICODE_WIDE
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
1914 *p++ = (Py_UNICODE)ch;
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
1915 #else
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1916 /* compute and append the two surrogates: */
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1917
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1918 /* translate from 10000..10FFFF to 0..FFFF */
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1919 ch -= 0x10000;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1920
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1921 /* high surrogate = top 10 bits added to D800 */
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1922 *p++ = (Py_UNICODE)(0xD800 + (ch >> 10));
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1923
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
1924 /* low surrogate = bottom 10 bits added to DC00 */
18279
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
1925 *p++ = (Py_UNICODE)(0xDC00 + (ch & 0x03FF));
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
1926 #endif
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1927 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1928
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1929 default:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1930 /* Other sizes are only needed for UCS-4 */
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1931 errmsg = "unsupported Unicode code range";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1932 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1933 endinpos = startinpos+n;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1934 goto utf8Error;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1935 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1936 s += n;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1937 continue;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
1938
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
1939 utf8Error:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1940 outpos = p-PyUnicode_AS_UNICODE(unicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1941 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1942 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1943 "utf8", errmsg,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1944 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1945 (PyObject **)&unicode, &outpos, &p))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1946 goto onError;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1947 }
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1948 if (consumed)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
1949 *consumed = s-starts;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1950
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1951 /* Adjust length */
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
1952 if (_PyUnicode_Resize(&unicode, p - unicode->str) < 0)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1953 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1954
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1955 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1956 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1957 return (PyObject *)unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1958
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1959 onError:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1960 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
1961 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1962 Py_DECREF(unicode);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1963 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1964 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1965
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1966 /* Allocation strategy: if the string is short, convert into a stack buffer
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1967 and allocate exactly as much space needed at the end. Else allocate the
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1968 maximum possible needed (4 result bytes per Unicode character), and return
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1969 the excess memory at the end.
23203
583c226afde6 Back out 2.140.
Martin v. Löwis <martin@v.loewis.de>
parents: 23198
diff changeset
1970 */
23198
950a22eb8cc8 PyUnicode_EncodeUTF8: squash compiler wng. The difference of two
Tim Peters <tim.peters@gmail.com>
parents: 23192
diff changeset
1971 PyObject *
950a22eb8cc8 PyUnicode_EncodeUTF8: squash compiler wng. The difference of two
Tim Peters <tim.peters@gmail.com>
parents: 23192
diff changeset
1972 PyUnicode_EncodeUTF8(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1973 Py_ssize_t size,
23198
950a22eb8cc8 PyUnicode_EncodeUTF8: squash compiler wng. The difference of two
Tim Peters <tim.peters@gmail.com>
parents: 23192
diff changeset
1974 const char *errors)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1975 {
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1976 #define MAX_SHORT_UNICHARS 300 /* largest size we'll do on the stack */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1977
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1978 Py_ssize_t i; /* index into s of next input byte */
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1979 PyObject *v; /* result string object */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1980 char *p; /* next free byte in output buffer */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
1981 Py_ssize_t nallocated; /* number of result bytes allocated */
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
1982 Py_ssize_t nneeded; /* number of result bytes needed */
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1983 char stackbuf[MAX_SHORT_UNICHARS * 4];
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1984
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1985 assert(s != NULL);
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1986 assert(size >= 0);
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1987
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1988 if (size <= MAX_SHORT_UNICHARS) {
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1989 /* Write into the stack buffer; nallocated can't overflow.
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1990 * At the end, we'll allocate exactly as much heap space as it
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1991 * turns out we need.
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1992 */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1993 nallocated = Py_SAFE_DOWNCAST(sizeof(stackbuf), size_t, int);
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1994 v = NULL; /* will allocate after we're done */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1995 p = stackbuf;
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1996 }
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1997 else {
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1998 /* Overallocate on the heap, and give the excess back at the end. */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
1999 nallocated = size * 4;
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2000 if (nallocated / 4 != size) /* overflow! */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2001 return PyErr_NoMemory();
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2002 v = PyString_FromStringAndSize(NULL, nallocated);
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2003 if (v == NULL)
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2004 return NULL;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2005 p = PyString_AS_STRING(v);
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2006 }
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2007
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2008 for (i = 0; i < size;) {
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
2009 Py_UCS4 ch = s[i++];
21882
94bac88f150f Fix for the UTF-8 memory allocation bug and the UTF-8 encoding
Marc-André Lemburg <mal@egenix.com>
parents: 21262
diff changeset
2010
23203
583c226afde6 Back out 2.140.
Martin v. Löwis <martin@v.loewis.de>
parents: 23198
diff changeset
2011 if (ch < 0x80)
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2012 /* Encode ASCII */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2013 *p++ = (char) ch;
21882
94bac88f150f Fix for the UTF-8 memory allocation bug and the UTF-8 encoding
Marc-André Lemburg <mal@egenix.com>
parents: 21262
diff changeset
2014
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2015 else if (ch < 0x0800) {
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2016 /* Encode Latin-1 */
21884
04cf4e13e5c9 Cosmetics.
Marc-André Lemburg <mal@egenix.com>
parents: 21883
diff changeset
2017 *p++ = (char)(0xc0 | (ch >> 6));
04cf4e13e5c9 Cosmetics.
Marc-André Lemburg <mal@egenix.com>
parents: 21883
diff changeset
2018 *p++ = (char)(0x80 | (ch & 0x3f));
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
2019 }
21882
94bac88f150f Fix for the UTF-8 memory allocation bug and the UTF-8 encoding
Marc-André Lemburg <mal@egenix.com>
parents: 21262
diff changeset
2020 else {
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2021 /* Encode UCS2 Unicode ordinals */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2022 if (ch < 0x10000) {
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2023 /* Special case: check for high surrogate */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2024 if (0xD800 <= ch && ch <= 0xDBFF && i != size) {
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2025 Py_UCS4 ch2 = s[i];
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2026 /* Check for low surrogate and combine the two to
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2027 form a UCS4 value */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2028 if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
23203
583c226afde6 Back out 2.140.
Martin v. Löwis <martin@v.loewis.de>
parents: 23198
diff changeset
2029 ch = ((ch - 0xD800) << 10 | (ch2 - 0xDC00)) + 0x10000;
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2030 i++;
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2031 goto encodeUCS4;
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
2032 }
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2033 /* Fall through: handles isolated high surrogates */
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
2034 }
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
2035 *p++ = (char)(0xe0 | (ch >> 12));
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2036 *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2037 *p++ = (char)(0x80 | (ch & 0x3f));
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2038 continue;
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2039 }
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2040 encodeUCS4:
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2041 /* Encode UCS4 Unicode ordinals */
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2042 *p++ = (char)(0xf0 | (ch >> 18));
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2043 *p++ = (char)(0x80 | ((ch >> 12) & 0x3f));
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2044 *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2045 *p++ = (char)(0x80 | (ch & 0x3f));
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2046 }
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2047 }
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2048
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2049 if (v == NULL) {
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2050 /* This was stack allocated. */
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
2051 nneeded = p - stackbuf;
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2052 assert(nneeded <= nallocated);
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2053 v = PyString_FromStringAndSize(stackbuf, nneeded);
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2054 }
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2055 else {
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2056 /* Cut back to size actually needed. */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2057 nneeded = p - PyString_AS_STRING(v);
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2058 assert(nneeded <= nallocated);
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2059 _PyString_Resize(&v, nneeded);
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2060 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2061 return v;
23203
583c226afde6 Back out 2.140.
Martin v. Löwis <martin@v.loewis.de>
parents: 23198
diff changeset
2062
23294
b735e89ccda7 SF patch 549375: Compromise PyUnicode_EncodeUTF8
Tim Peters <tim.peters@gmail.com>
parents: 23228
diff changeset
2063 #undef MAX_SHORT_UNICHARS
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2064 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2065
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2066 PyObject *PyUnicode_AsUTF8String(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2067 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2068 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2069 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2070 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2071 }
14059
d0348a26b093 PyUnicode_AsUTF8String(): Don't need to explicitly incref str since
Barry Warsaw <barry@python.org>
parents: 13966
diff changeset
2072 return PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(unicode),
d0348a26b093 PyUnicode_AsUTF8String(): Don't need to explicitly incref str since
Barry Warsaw <barry@python.org>
parents: 13966
diff changeset
2073 PyUnicode_GET_SIZE(unicode),
d0348a26b093 PyUnicode_AsUTF8String(): Don't need to explicitly incref str since
Barry Warsaw <barry@python.org>
parents: 13966
diff changeset
2074 NULL);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2075 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2076
42649
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2077 /* --- UTF-32 Codec ------------------------------------------------------- */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2078
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2079 PyObject *
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2080 PyUnicode_DecodeUTF32(const char *s,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2081 Py_ssize_t size,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2082 const char *errors,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2083 int *byteorder)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2084 {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2085 return PyUnicode_DecodeUTF32Stateful(s, size, errors, byteorder, NULL);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2086 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2087
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2088 PyObject *
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2089 PyUnicode_DecodeUTF32Stateful(const char *s,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2090 Py_ssize_t size,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2091 const char *errors,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2092 int *byteorder,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2093 Py_ssize_t *consumed)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2094 {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2095 const char *starts = s;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2096 Py_ssize_t startinpos;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2097 Py_ssize_t endinpos;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2098 Py_ssize_t outpos;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2099 PyUnicodeObject *unicode;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2100 Py_UNICODE *p;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2101 #ifndef Py_UNICODE_WIDE
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2102 int i, pairs;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2103 #else
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2104 const int pairs = 0;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2105 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2106 const unsigned char *q, *e;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2107 int bo = 0; /* assume native ordering by default */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2108 const char *errmsg = "";
42650
f2f0e95a797b Move variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42649
diff changeset
2109 /* Offsets from q for retrieving bytes in the right order. */
f2f0e95a797b Move variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42649
diff changeset
2110 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
f2f0e95a797b Move variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42649
diff changeset
2111 int iorder[] = {0, 1, 2, 3};
f2f0e95a797b Move variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42649
diff changeset
2112 #else
f2f0e95a797b Move variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42649
diff changeset
2113 int iorder[] = {3, 2, 1, 0};
f2f0e95a797b Move variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42649
diff changeset
2114 #endif
42652
9c4f5b1d6b9d Move another variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42650
diff changeset
2115 PyObject *errorHandler = NULL;
9c4f5b1d6b9d Move another variable declaration up.
Walter Dörwald <walter@livinglogic.de>
parents: 42650
diff changeset
2116 PyObject *exc = NULL;
42649
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2117 /* On narrow builds we split characters outside the BMP into two
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2118 codepoints => count how much extra space we need. */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2119 #ifndef Py_UNICODE_WIDE
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2120 for (i = pairs = 0; i < size/4; i++)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2121 if (((Py_UCS4 *)s)[i] >= 0x10000)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2122 pairs++;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2123 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2124
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2125 /* This might be one to much, because of a BOM */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2126 unicode = _PyUnicode_New((size+3)/4+pairs);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2127 if (!unicode)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2128 return NULL;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2129 if (size == 0)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2130 return (PyObject *)unicode;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2131
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2132 /* Unpack UTF-32 encoded data */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2133 p = unicode->str;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2134 q = (unsigned char *)s;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2135 e = q + size;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2136
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2137 if (byteorder)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2138 bo = *byteorder;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2139
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2140 /* Check for BOM marks (U+FEFF) in the input and adjust current
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2141 byte order setting accordingly. In native mode, the leading BOM
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2142 mark is skipped, in all other modes, it is copied to the output
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2143 stream as-is (giving a ZWNBSP character). */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2144 if (bo == 0) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2145 if (size >= 4) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2146 const Py_UCS4 bom = (q[iorder[3]] << 24) | (q[iorder[2]] << 16) |
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2147 (q[iorder[1]] << 8) | q[iorder[0]];
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2148 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2149 if (bom == 0x0000FEFF) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2150 q += 4;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2151 bo = -1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2152 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2153 else if (bom == 0xFFFE0000) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2154 q += 4;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2155 bo = 1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2156 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2157 #else
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2158 if (bom == 0x0000FEFF) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2159 q += 4;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2160 bo = 1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2161 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2162 else if (bom == 0xFFFE0000) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2163 q += 4;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2164 bo = -1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2165 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2166 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2167 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2168 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2169
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2170 if (bo == -1) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2171 /* force LE */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2172 iorder[0] = 0;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2173 iorder[1] = 1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2174 iorder[2] = 2;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2175 iorder[3] = 3;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2176 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2177 else if (bo == 1) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2178 /* force BE */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2179 iorder[0] = 3;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2180 iorder[1] = 2;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2181 iorder[2] = 1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2182 iorder[3] = 0;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2183 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2184
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2185 while (q < e) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2186 Py_UCS4 ch;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2187 /* remaining bytes at the end? (size should be divisible by 4) */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2188 if (e-q<4) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2189 if (consumed)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2190 break;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2191 errmsg = "truncated data";
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2192 startinpos = ((const char *)q)-starts;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2193 endinpos = ((const char *)e)-starts;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2194 goto utf32Error;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2195 /* The remaining input chars are ignored if the callback
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2196 chooses to skip the input */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2197 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2198 ch = (q[iorder[3]] << 24) | (q[iorder[2]] << 16) |
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2199 (q[iorder[1]] << 8) | q[iorder[0]];
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2200
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2201 if (ch >= 0x110000)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2202 {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2203 errmsg = "codepoint not in range(0x110000)";
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2204 startinpos = ((const char *)q)-starts;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2205 endinpos = startinpos+4;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2206 goto utf32Error;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2207 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2208 #ifndef Py_UNICODE_WIDE
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2209 if (ch >= 0x10000)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2210 {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2211 *p++ = 0xD800 | ((ch-0x10000) >> 10);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2212 *p++ = 0xDC00 | ((ch-0x10000) & 0x3FF);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2213 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2214 else
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2215 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2216 *p++ = ch;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2217 q += 4;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2218 continue;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2219 utf32Error:
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2220 outpos = p-PyUnicode_AS_UNICODE(unicode);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2221 if (unicode_decode_call_errorhandler(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2222 errors, &errorHandler,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2223 "utf32", errmsg,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2224 starts, size, &startinpos, &endinpos, &exc, &s,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2225 (PyObject **)&unicode, &outpos, &p))
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2226 goto onError;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2227 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2228
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2229 if (byteorder)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2230 *byteorder = bo;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2231
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2232 if (consumed)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2233 *consumed = (const char *)q-starts;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2234
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2235 /* Adjust length */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2236 if (_PyUnicode_Resize(&unicode, p - unicode->str) < 0)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2237 goto onError;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2238
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2239 Py_XDECREF(errorHandler);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2240 Py_XDECREF(exc);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2241 return (PyObject *)unicode;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2242
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2243 onError:
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2244 Py_DECREF(unicode);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2245 Py_XDECREF(errorHandler);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2246 Py_XDECREF(exc);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2247 return NULL;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2248 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2249
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2250 PyObject *
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2251 PyUnicode_EncodeUTF32(const Py_UNICODE *s,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2252 Py_ssize_t size,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2253 const char *errors,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2254 int byteorder)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2255 {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2256 PyObject *v;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2257 unsigned char *p;
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2258 Py_ssize_t nsize, bytesize;
42649
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2259 #ifndef Py_UNICODE_WIDE
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2260 Py_ssize_t i, pairs;
42649
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2261 #else
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2262 const int pairs = 0;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2263 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2264 /* Offsets from p for storing byte pairs in the right order. */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2265 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2266 int iorder[] = {0, 1, 2, 3};
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2267 #else
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2268 int iorder[] = {3, 2, 1, 0};
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2269 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2270
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2271 #define STORECHAR(CH) \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2272 do { \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2273 p[iorder[3]] = ((CH) >> 24) & 0xff; \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2274 p[iorder[2]] = ((CH) >> 16) & 0xff; \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2275 p[iorder[1]] = ((CH) >> 8) & 0xff; \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2276 p[iorder[0]] = (CH) & 0xff; \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2277 p += 4; \
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2278 } while(0)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2279
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2280 /* In narrow builds we can output surrogate pairs as one codepoint,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2281 so we need less space. */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2282 #ifndef Py_UNICODE_WIDE
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2283 for (i = pairs = 0; i < size-1; i++)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2284 if (0xD800 <= s[i] && s[i] <= 0xDBFF &&
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2285 0xDC00 <= s[i+1] && s[i+1] <= 0xDFFF)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2286 pairs++;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2287 #endif
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2288 nsize = (size - pairs + (byteorder == 0));
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2289 bytesize = nsize * 4;
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2290 if (bytesize / 4 != nsize)
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2291 return PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2292 v = PyString_FromStringAndSize(NULL, bytesize);
42649
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2293 if (v == NULL)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2294 return NULL;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2295
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2296 p = (unsigned char *)PyString_AS_STRING(v);
42649
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2297 if (byteorder == 0)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2298 STORECHAR(0xFEFF);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2299 if (size == 0)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2300 return v;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2301
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2302 if (byteorder == -1) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2303 /* force LE */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2304 iorder[0] = 0;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2305 iorder[1] = 1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2306 iorder[2] = 2;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2307 iorder[3] = 3;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2308 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2309 else if (byteorder == 1) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2310 /* force BE */
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2311 iorder[0] = 3;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2312 iorder[1] = 2;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2313 iorder[2] = 1;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2314 iorder[3] = 0;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2315 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2316
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2317 while (size-- > 0) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2318 Py_UCS4 ch = *s++;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2319 #ifndef Py_UNICODE_WIDE
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2320 if (0xD800 <= ch && ch <= 0xDBFF && size > 0) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2321 Py_UCS4 ch2 = *s;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2322 if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2323 ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2324 s++;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2325 size--;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2326 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2327 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2328 #endif
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2329 STORECHAR(ch);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2330 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2331 return v;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2332 #undef STORECHAR
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2333 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2334
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2335 PyObject *PyUnicode_AsUTF32String(PyObject *unicode)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2336 {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2337 if (!PyUnicode_Check(unicode)) {
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2338 PyErr_BadArgument();
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2339 return NULL;
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2340 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2341 return PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(unicode),
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2342 PyUnicode_GET_SIZE(unicode),
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2343 NULL,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2344 0);
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2345 }
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs.
Walter Dörwald <walter@livinglogic.de>
parents: 42402
diff changeset
2346
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2347 /* --- UTF-16 Codec ------------------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2348
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2349 PyObject *
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2350 PyUnicode_DecodeUTF16(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2351 Py_ssize_t size,
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2352 const char *errors,
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2353 int *byteorder)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2354 {
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2355 return PyUnicode_DecodeUTF16Stateful(s, size, errors, byteorder, NULL);
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2356 }
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2357
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2358 PyObject *
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2359 PyUnicode_DecodeUTF16Stateful(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2360 Py_ssize_t size,
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2361 const char *errors,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2362 int *byteorder,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2363 Py_ssize_t *consumed)
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2364 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2365 const char *starts = s;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2366 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2367 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2368 Py_ssize_t outpos;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2369 PyUnicodeObject *unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2370 Py_UNICODE *p;
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2371 const unsigned char *q, *e;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2372 int bo = 0; /* assume native ordering by default */
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2373 const char *errmsg = "";
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2374 /* Offsets from q for retrieving byte pairs in the right order. */
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2375 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2376 int ihi = 1, ilo = 0;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2377 #else
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2378 int ihi = 0, ilo = 1;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2379 #endif
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2380 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2381 PyObject *exc = NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2382
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2383 /* Note: size will always be longer than the resulting Unicode
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2384 character count */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2385 unicode = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2386 if (!unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2387 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2388 if (size == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2389 return (PyObject *)unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2390
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2391 /* Unpack UTF-16 encoded data */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2392 p = unicode->str;
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2393 q = (unsigned char *)s;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2394 e = q + size;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2395
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2396 if (byteorder)
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2397 bo = *byteorder;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2398
17887
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2399 /* Check for BOM marks (U+FEFF) in the input and adjust current
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2400 byte order setting accordingly. In native mode, the leading BOM
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2401 mark is skipped, in all other modes, it is copied to the output
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2402 stream as-is (giving a ZWNBSP character). */
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2403 if (bo == 0) {
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2404 if (size >= 2) {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2405 const Py_UNICODE bom = (q[ihi] << 8) | q[ilo];
17887
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2406 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2407 if (bom == 0xFEFF) {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2408 q += 2;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2409 bo = -1;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2410 }
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2411 else if (bom == 0xFFFE) {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2412 q += 2;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2413 bo = 1;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2414 }
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2415 #else
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2416 if (bom == 0xFEFF) {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2417 q += 2;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2418 bo = 1;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2419 }
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2420 else if (bom == 0xFFFE) {
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2421 q += 2;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2422 bo = -1;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2423 }
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2424 #endif
17887
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2425 }
fd5f83188778 This patch changes the behaviour of the UTF-16 codec family. Only the
Marc-André Lemburg <mal@egenix.com>
parents: 17700
diff changeset
2426 }
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2427
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2428 if (bo == -1) {
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2429 /* force LE */
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2430 ihi = 1;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2431 ilo = 0;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2432 }
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2433 else if (bo == 1) {
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2434 /* force BE */
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2435 ihi = 0;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2436 ilo = 1;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2437 }
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2438
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2439 while (q < e) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2440 Py_UNICODE ch;
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2441 /* remaining bytes at the end? (size should be even) */
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2442 if (e-q<2) {
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2443 if (consumed)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2444 break;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2445 errmsg = "truncated data";
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2446 startinpos = ((const char *)q)-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2447 endinpos = ((const char *)e)-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2448 goto utf16Error;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2449 /* The remaining input chars are ignored if the callback
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2450 chooses to skip the input */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2451 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2452 ch = (q[ihi] << 8) | q[ilo];
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2453
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2454 q += 2;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2455
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2456 if (ch < 0xD800 || ch > 0xDFFF) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2457 *p++ = ch;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2458 continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2459 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2460
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2461 /* UTF-16 code pair: */
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2462 if (q >= e) {
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2463 errmsg = "unexpected end of data";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2464 startinpos = (((const char *)q)-2)-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2465 endinpos = ((const char *)e)-starts;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2466 goto utf16Error;
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2467 }
18295
20e3a6e10c24 When decoding UTF-16, don't assume that the buffer is in native endianness
Martin v. Löwis <martin@v.loewis.de>
parents: 18292
diff changeset
2468 if (0xD800 <= ch && ch <= 0xDBFF) {
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2469 Py_UNICODE ch2 = (q[ihi] << 8) | q[ilo];
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2470 q += 2;
18295
20e3a6e10c24 When decoding UTF-16, don't assume that the buffer is in native endianness
Martin v. Löwis <martin@v.loewis.de>
parents: 18292
diff changeset
2471 if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
18324
3ba7a358ce02 use Py_UNICODE_WIDE instead of USE_UCS4_STORAGE and Py_UNICODE_SIZE
Fredrik Lundh <fredrik@pythonware.com>
parents: 18298
diff changeset
2472 #ifndef Py_UNICODE_WIDE
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2473 *p++ = ch;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2474 *p++ = ch2;
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2475 #else
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2476 *p++ = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000;
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2477 #endif
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2478 continue;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2479 }
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2480 else {
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2481 errmsg = "illegal UTF-16 surrogate";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2482 startinpos = (((const char *)q)-4)-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2483 endinpos = startinpos+2;
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2484 goto utf16Error;
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2485 }
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2486
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2487 }
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2488 errmsg = "illegal encoding";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2489 startinpos = (((const char *)q)-2)-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2490 endinpos = startinpos+2;
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2491 /* Fall through to report the error */
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2492
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2493 utf16Error:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2494 outpos = p-PyUnicode_AS_UNICODE(unicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2495 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2496 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2497 "utf16", errmsg,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2498 starts, size, &startinpos, &endinpos, &exc, (const char **)&q,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2499 (PyObject **)&unicode, &outpos, &p))
13635
dfb7ecfffe3d Fixed problems with UTF error reporting macros and some formatting bugs.
Marc-André Lemburg <mal@egenix.com>
parents: 13633
diff changeset
2500 goto onError;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2501 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2502
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2503 if (byteorder)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2504 *byteorder = bo;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2505
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2506 if (consumed)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2507 *consumed = (const char *)q-starts;
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 33311
diff changeset
2508
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2509 /* Adjust length */
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
2510 if (_PyUnicode_Resize(&unicode, p - unicode->str) < 0)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2511 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2512
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2513 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2514 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2515 return (PyObject *)unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2516
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2517 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2518 Py_DECREF(unicode);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2519 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2520 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2521 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2522 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2523
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2524 PyObject *
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2525 PyUnicode_EncodeUTF16(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2526 Py_ssize_t size,
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2527 const char *errors,
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2528 int byteorder)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2529 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2530 PyObject *v;
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2531 unsigned char *p;
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2532 Py_ssize_t nsize, bytesize;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2533 #ifdef Py_UNICODE_WIDE
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2534 Py_ssize_t i, pairs;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2535 #else
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2536 const int pairs = 0;
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2537 #endif
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2538 /* Offsets from p for storing byte pairs in the right order. */
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2539 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2540 int ihi = 1, ilo = 0;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2541 #else
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2542 int ihi = 0, ilo = 1;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2543 #endif
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2544
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2545 #define STORECHAR(CH) \
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2546 do { \
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2547 p[ihi] = ((CH) >> 8) & 0xff; \
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2548 p[ilo] = (CH) & 0xff; \
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2549 p += 2; \
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2550 } while(0)
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2551
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2552 #ifdef Py_UNICODE_WIDE
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2553 for (i = pairs = 0; i < size; i++)
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2554 if (s[i] >= 0x10000)
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2555 pairs++;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2556 #endif
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2557 /* 2 * (size + pairs + (byteorder == 0)) */
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2558 if (size > PY_SSIZE_T_MAX ||
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2559 size > PY_SSIZE_T_MAX - pairs - (byteorder == 0))
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2560 return PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2561 nsize = size + pairs + (byteorder == 0);
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2562 bytesize = nsize * 2;
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2563 if (bytesize / 2 != nsize)
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2564 return PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2565 v = PyString_FromStringAndSize(NULL, bytesize);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2566 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2567 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2568
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2569 p = (unsigned char *)PyString_AS_STRING(v);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2570 if (byteorder == 0)
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2571 STORECHAR(0xFEFF);
13396
759f698740cc Fix to bug #393 (UTF16 codec didn't like empty strings) and
Marc-André Lemburg <mal@egenix.com>
parents: 13394
diff changeset
2572 if (size == 0)
13966
91ddeed12fa7 Fixed a couple of instances where a 0-length string was being
Marc-André Lemburg <mal@egenix.com>
parents: 13946
diff changeset
2573 return v;
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2574
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2575 if (byteorder == -1) {
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2576 /* force LE */
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2577 ihi = 1;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2578 ilo = 0;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2579 }
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2580 else if (byteorder == 1) {
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2581 /* force BE */
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2582 ihi = 0;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2583 ilo = 1;
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2584 }
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2585
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2586 while (size-- > 0) {
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2587 Py_UNICODE ch = *s++;
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2588 Py_UNICODE ch2 = 0;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2589 #ifdef Py_UNICODE_WIDE
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2590 if (ch >= 0x10000) {
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2591 ch2 = 0xDC00 | ((ch-0x10000) & 0x3FF);
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2592 ch = 0xD800 | ((ch-0x10000) >> 10);
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2593 }
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
2594 #endif
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2595 STORECHAR(ch);
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2596 if (ch2)
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2597 STORECHAR(ch2);
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2598 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2599 return v;
19121
77e405f3a942 SF patch #438013 Remove 2-byte Py_UCS2 assumptions
Tim Peters <tim.peters@gmail.com>
parents: 18933
diff changeset
2600 #undef STORECHAR
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2601 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2602
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2603 PyObject *PyUnicode_AsUTF16String(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2604 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2605 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2606 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2607 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2608 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2609 return PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2610 PyUnicode_GET_SIZE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2611 NULL,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2612 0);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2613 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2614
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2615 /* --- Unicode Escape Codec ----------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2616
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16064
diff changeset
2617 static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL;
13063
eb243443b884 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13049
diff changeset
2618
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2619 PyObject *PyUnicode_DecodeUnicodeEscape(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2620 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2621 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2622 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2623 const char *starts = s;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2624 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2625 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2626 Py_ssize_t outpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2627 int i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2628 PyUnicodeObject *v;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2629 Py_UNICODE *p;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2630 const char *end;
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2631 char* message;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2632 Py_UCS4 chr = 0xffffffff; /* in case 'getcode' messes up */
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2633 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2634 PyObject *exc = NULL;
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2635
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2636 /* Escaped strings will always be longer than the resulting
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2637 Unicode string, so we start with size here and then reduce the
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2638 length after conversion to the true value.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2639 (but if the error callback returns a long replacement string
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2640 we'll have to allocate more space) */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2641 v = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2642 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2643 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2644 if (size == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2645 return (PyObject *)v;
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2646
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2647 p = PyUnicode_AS_UNICODE(v);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2648 end = s + size;
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2649
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2650 while (s < end) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2651 unsigned char c;
13396
759f698740cc Fix to bug #393 (UTF16 codec didn't like empty strings) and
Marc-André Lemburg <mal@egenix.com>
parents: 13394
diff changeset
2652 Py_UNICODE x;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2653 int digits;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2654
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2655 /* Non-escape characters are interpreted as Unicode ordinals */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2656 if (*s != '\\') {
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2657 *p++ = (unsigned char) *s++;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2658 continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2659 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2660
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2661 startinpos = s-starts;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2662 /* \ - Escapes */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2663 s++;
43710
124cbf1a0e3e Backport fixes for the code that decodes octal escapes (and for PyString
Guido van Rossum <guido@python.org>
parents: 43053
diff changeset
2664 c = *s++;
124cbf1a0e3e Backport fixes for the code that decodes octal escapes (and for PyString
Guido van Rossum <guido@python.org>
parents: 43053
diff changeset
2665 if (s > end)
124cbf1a0e3e Backport fixes for the code that decodes octal escapes (and for PyString
Guido van Rossum <guido@python.org>
parents: 43053
diff changeset
2666 c = '\0'; /* Invalid after \ */
124cbf1a0e3e Backport fixes for the code that decodes octal escapes (and for PyString
Guido van Rossum <guido@python.org>
parents: 43053
diff changeset
2667 switch (c) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2668
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2669 /* \x escapes */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2670 case '\n': break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2671 case '\\': *p++ = '\\'; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2672 case '\'': *p++ = '\''; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2673 case '\"': *p++ = '\"'; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2674 case 'b': *p++ = '\b'; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2675 case 'f': *p++ = '\014'; break; /* FF */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2676 case 't': *p++ = '\t'; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2677 case 'n': *p++ = '\n'; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2678 case 'r': *p++ = '\r'; break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2679 case 'v': *p++ = '\013'; break; /* VT */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2680 case 'a': *p++ = '\007'; break; /* BEL, not classic C */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2681
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2682 /* \OOO (octal) escapes */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2683 case '0': case '1': case '2': case '3':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2684 case '4': case '5': case '6': case '7':
12544
599bfa4ffd49 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12511
diff changeset
2685 x = s[-1] - '0';
43710
124cbf1a0e3e Backport fixes for the code that decodes octal escapes (and for PyString
Guido van Rossum <guido@python.org>
parents: 43053
diff changeset
2686 if (s < end && '0' <= *s && *s <= '7') {
12544
599bfa4ffd49 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12511
diff changeset
2687 x = (x<<3) + *s++ - '0';
43710
124cbf1a0e3e Backport fixes for the code that decodes octal escapes (and for PyString
Guido van Rossum <guido@python.org>
parents: 43053
diff changeset
2688 if (s < end && '0' <= *s && *s <= '7')
12544
599bfa4ffd49 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12511
diff changeset
2689 x = (x<<3) + *s++ - '0';
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2690 }
12544
599bfa4ffd49 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12511
diff changeset
2691 *p++ = x;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2692 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2693
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2694 /* hex escapes */
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2695 /* \xXX */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2696 case 'x':
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2697 digits = 2;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2698 message = "truncated \\xXX escape";
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2699 goto hexescape;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2700
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2701 /* \uXXXX */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2702 case 'u':
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2703 digits = 4;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2704 message = "truncated \\uXXXX escape";
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2705 goto hexescape;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2706
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2707 /* \UXXXXXXXX */
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2708 case 'U':
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2709 digits = 8;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2710 message = "truncated \\UXXXXXXXX escape";
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2711 hexescape:
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2712 chr = 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2713 outpos = p-PyUnicode_AS_UNICODE(v);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2714 if (s+digits>end) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2715 endinpos = size;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2716 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2717 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2718 "unicodeescape", "end of string in escape sequence",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2719 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2720 (PyObject **)&v, &outpos, &p))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2721 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2722 goto nextByte;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2723 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2724 for (i = 0; i < digits; ++i) {
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2725 c = (unsigned char) s[i];
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2726 if (!isxdigit(c)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2727 endinpos = (s+i+1)-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2728 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2729 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2730 "unicodeescape", message,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2731 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2732 (PyObject **)&v, &outpos, &p))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2733 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2734 goto nextByte;
14388
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2735 }
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2736 chr = (chr<<4) & ~0xF;
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2737 if (c >= '0' && c <= '9')
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2738 chr += c - '0';
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2739 else if (c >= 'a' && c <= 'f')
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2740 chr += 10 + c - 'a';
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2741 else
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2742 chr += 10 + c - 'A';
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2743 }
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2744 s += i;
30653
3b47d6057151 Fix for SF bug [ 817156 ] invalid \U escape gives 0=length unistr.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30441
diff changeset
2745 if (chr == 0xffffffff && PyErr_Occurred())
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2746 /* _decoding_error will have already written into the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2747 target buffer. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2748 break;
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2749 store:
14388
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2750 /* when we get here, chr is a 32-bit unicode character */
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2751 if (chr <= 0xffff)
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2752 /* UCS-2 character */
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2753 *p++ = (Py_UNICODE) chr;
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2754 else if (chr <= 0x10ffff) {
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2755 /* UCS-4 character. Either store directly, or as
22521
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2756 surrogate pair. */
18324
3ba7a358ce02 use Py_UNICODE_WIDE instead of USE_UCS4_STORAGE and Py_UNICODE_SIZE
Fredrik Lundh <fredrik@pythonware.com>
parents: 18298
diff changeset
2757 #ifdef Py_UNICODE_WIDE
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2758 *p++ = chr;
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2759 #else
14388
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2760 chr -= 0x10000L;
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2761 *p++ = 0xD800 + (Py_UNICODE) (chr >> 10);
18279
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
2762 *p++ = 0xDC00 + (Py_UNICODE) (chr & 0x03FF);
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2763 #endif
14388
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2764 } else {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2765 endinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2766 outpos = p-PyUnicode_AS_UNICODE(v);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2767 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2768 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2769 "unicodeescape", "illegal Unicode character",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2770 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2771 (PyObject **)&v, &outpos, &p))
14388
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2772 goto onError;
ef2714bb0730 changed \x to consume exactly two hex digits, also for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 14067
diff changeset
2773 }
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2774 break;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2775
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2776 /* \N{name} */
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2777 case 'N':
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2778 message = "malformed \\N character escape";
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2779 if (ucnhash_CAPI == NULL) {
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2780 /* load the unicode data module */
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
2781 PyObject *m, *api;
44439
0cbe94bea2c7 Modified PyImport_Import and PyImport_ImportModule to always use absolute imports by calling __import__ with an explicit level of 0
Christian Heimes <christian@cheimes.de>
parents: 44350
diff changeset
2782 m = PyImport_ImportModuleNoBlock("unicodedata");
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2783 if (m == NULL)
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2784 goto ucnhashError;
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
2785 api = PyObject_GetAttrString(m, "ucnhash_CAPI");
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2786 Py_DECREF(m);
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
2787 if (api == NULL)
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2788 goto ucnhashError;
37596
1b078493d538 More low-hanging fruit. Still need to re-arrange some code (or find a better
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37306
diff changeset
2789 ucnhash_CAPI = (_PyUnicode_Name_CAPI *)PyCObject_AsVoidPtr(api);
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
2790 Py_DECREF(api);
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2791 if (ucnhash_CAPI == NULL)
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2792 goto ucnhashError;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2793 }
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2794 if (*s == '{') {
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2795 const char *start = s+1;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2796 /* look for the closing brace */
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2797 while (*s != '}' && s < end)
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2798 s++;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2799 if (s > start && s < end && *s == '}') {
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2800 /* found a name. look it up in the unicode database */
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2801 message = "unknown Unicode character name";
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2802 s++;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36882
diff changeset
2803 if (ucnhash_CAPI->getcode(NULL, start, (int)(s-start-1), &chr))
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2804 goto store;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2805 }
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2806 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2807 endinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2808 outpos = p-PyUnicode_AS_UNICODE(v);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2809 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2810 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2811 "unicodeescape", message,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2812 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2813 (PyObject **)&v, &outpos, &p))
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2814 goto onError;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2815 break;
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2816
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2817 default:
22521
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2818 if (s > end) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2819 message = "\\ at end of string";
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2820 s--;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2821 endinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2822 outpos = p-PyUnicode_AS_UNICODE(v);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2823 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2824 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2825 "unicodeescape", message,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2826 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2827 (PyObject **)&v, &outpos, &p))
22521
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2828 goto onError;
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2829 }
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2830 else {
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2831 *p++ = '\\';
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2832 *p++ = (unsigned char)s[-1];
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2833 }
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2834 break;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2835 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2836 nextByte:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2837 ;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2838 }
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
2839 if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2840 goto onError;
30160
3329b058db20 Fix another refcounting leak (in PyUnicode_DecodeUnicodeEscape()).
Walter Dörwald <walter@livinglogic.de>
parents: 30149
diff changeset
2841 Py_XDECREF(errorHandler);
3329b058db20 Fix another refcounting leak (in PyUnicode_DecodeUnicodeEscape()).
Walter Dörwald <walter@livinglogic.de>
parents: 30149
diff changeset
2842 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2843 return (PyObject *)v;
22521
795bdd692176 Fix whitespace.
Walter Dörwald <walter@livinglogic.de>
parents: 22446
diff changeset
2844
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2845 ucnhashError:
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16064
diff changeset
2846 PyErr_SetString(
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16064
diff changeset
2847 PyExc_UnicodeError,
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16064
diff changeset
2848 "\\N escapes not supported (can't load unicodedata module)"
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16064
diff changeset
2849 );
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
2850 Py_XDECREF(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2851 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2852 Py_XDECREF(exc);
16064
a97aa9238771 Better error message if ucnhash cannot be found (obscure attribute
Fredrik Lundh <fredrik@pythonware.com>
parents: 16029
diff changeset
2853 return NULL;
a97aa9238771 Better error message if ucnhash cannot be found (obscure attribute
Fredrik Lundh <fredrik@pythonware.com>
parents: 16029
diff changeset
2854
16667
725238293869 reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make it
Fredrik Lundh <fredrik@pythonware.com>
parents: 16332
diff changeset
2855 onError:
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2856 Py_XDECREF(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2857 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
2858 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2859 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2860 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2861
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2862 /* Return a Unicode-Escape string version of the Unicode object.
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2863
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2864 If quotes is true, the string is enclosed in u"" or u'' quotes as
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2865 appropriate.
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2866
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2867 */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2868
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
2869 Py_LOCAL_INLINE(const Py_UNICODE *) findchar(const Py_UNICODE *s,
38347
6d8615f9f1cc use Py_LOCAL also for string and unicode objects
Fredrik Lundh <fredrik@pythonware.com>
parents: 38339
diff changeset
2870 Py_ssize_t size,
6d8615f9f1cc use Py_LOCAL also for string and unicode objects
Fredrik Lundh <fredrik@pythonware.com>
parents: 38339
diff changeset
2871 Py_UNICODE ch)
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2872 {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2873 /* like wcschr, but doesn't stop at NULL characters */
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2874
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2875 while (size-- > 0) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2876 if (*s == ch)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2877 return s;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2878 s++;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2879 }
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2880
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2881 return NULL;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
2882 }
12025
7994d9db7f4d On 17-Mar-2000, Marc-Andre Lemburg said:
Barry Warsaw <barry@python.org>
parents: 11997
diff changeset
2883
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2884 static
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2885 PyObject *unicodeescape_string(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
2886 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2887 int quotes)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2888 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2889 PyObject *repr;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2890 char *p;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2891
16226
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
2892 static const char *hexdigit = "0123456789abcdef";
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2893 #ifdef Py_UNICODE_WIDE
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2894 const Py_ssize_t expandsize = 10;
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2895 #else
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2896 const Py_ssize_t expandsize = 6;
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2897 #endif
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2898
39621
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2899 /* XXX(nnorwitz): rather than over-allocating, it would be
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2900 better to choose a different scheme. Perhaps scan the
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2901 first N-chars of the string and allocate based on that size.
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2902 */
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2903 /* Initial allocation is based on the longest-possible unichr
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2904 escape.
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2905
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2906 In wide (UTF-32) builds '\U00xxxxxx' is 10 chars per source
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2907 unichr, so in this case it's the longest unichr escape. In
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2908 narrow (UTF-16) builds this is five chars per source unichr
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2909 since there are two unichrs in the surrogate pair, so in narrow
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2910 (UTF-16) builds it's not the longest unichr escape.
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2911
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2912 In wide or narrow builds '\uxxxx' is 6 chars per source unichr,
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2913 so in the narrow (UTF-16) build case it's the longest unichr
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2914 escape.
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2915 */
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2916
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2917 if (size > (PY_SSIZE_T_MAX - 2 - 1) / expandsize)
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2918 return PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2919
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2920 repr = PyString_FromStringAndSize(NULL,
39621
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2921 2
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
2922 + expandsize*size
39621
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2923 + 1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2924 if (repr == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2925 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2926
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2927 p = PyString_AS_STRING(repr);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2928
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2929 if (quotes) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2930 *p++ = 'u';
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
2931 *p++ = (findchar(s, size, '\'') &&
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2932 !findchar(s, size, '"')) ? '"' : '\'';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2933 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2934 while (size-- > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2935 Py_UNICODE ch = *s++;
18828
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
2936
36056
f1ae11457d34 Bug #1379994: Fix *unicode_escape codecs to encode r'\' as r'\\'
Hye-Shik Chang <hyeshik@gmail.com>
parents: 35999
diff changeset
2937 /* Escape quotes and backslashes */
f1ae11457d34 Bug #1379994: Fix *unicode_escape codecs to encode r'\' as r'\\'
Hye-Shik Chang <hyeshik@gmail.com>
parents: 35999
diff changeset
2938 if ((quotes &&
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
2939 ch == (Py_UNICODE) PyString_AS_STRING(repr)[1]) || ch == '\\') {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2940 *p++ = '\\';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2941 *p++ = (char) ch;
20018
8aa1ada33f64 Fix a bug in rendering of \\ by repr() -- it rendered as \\\ instead
Guido van Rossum <guido@python.org>
parents: 20007
diff changeset
2942 continue;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
2943 }
18828
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
2944
18773
743f8e1d953c #ifdef out generation of \U escapes unless Py_UNICODE_WIDE. This
Guido van Rossum <guido@python.org>
parents: 18324
diff changeset
2945 #ifdef Py_UNICODE_WIDE
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2946 /* Map 21-bit characters to '\U00xxxxxx' */
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2947 else if (ch >= 0x10000) {
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2948 *p++ = '\\';
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2949 *p++ = 'U';
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2950 *p++ = hexdigit[(ch >> 28) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2951 *p++ = hexdigit[(ch >> 24) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2952 *p++ = hexdigit[(ch >> 20) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2953 *p++ = hexdigit[(ch >> 16) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2954 *p++ = hexdigit[(ch >> 12) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2955 *p++ = hexdigit[(ch >> 8) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2956 *p++ = hexdigit[(ch >> 4) & 0x0000000F];
18828
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
2957 *p++ = hexdigit[ch & 0x0000000F];
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
2958 continue;
18292
ffbb9c918ac0 Support using UCS-4 as the Py_UNICODE type:
Martin v. Löwis <martin@v.loewis.de>
parents: 18280
diff changeset
2959 }
39621
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2960 #else
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2961 /* Map UTF-16 surrogate pairs to '\U00xxxxxx' */
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2962 else if (ch >= 0xD800 && ch < 0xDC00) {
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2963 Py_UNICODE ch2;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2964 Py_UCS4 ucs;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
2965
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2966 ch2 = *s++;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2967 size--;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2968 if (ch2 >= 0xDC00 && ch2 <= 0xDFFF) {
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2969 ucs = (((ch & 0x03FF) << 10) | (ch2 & 0x03FF)) + 0x00010000;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2970 *p++ = '\\';
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2971 *p++ = 'U';
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2972 *p++ = hexdigit[(ucs >> 28) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2973 *p++ = hexdigit[(ucs >> 24) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2974 *p++ = hexdigit[(ucs >> 20) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2975 *p++ = hexdigit[(ucs >> 16) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2976 *p++ = hexdigit[(ucs >> 12) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2977 *p++ = hexdigit[(ucs >> 8) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2978 *p++ = hexdigit[(ucs >> 4) & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2979 *p++ = hexdigit[ucs & 0x0000000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2980 continue;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2981 }
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2982 /* Fall through: isolated surrogates are copied as-is */
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2983 s--;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2984 size++;
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2985 }
39621
e8a65bbd6a29 Patch #1541585: fix buffer overrun when performing repr() on
Neal Norwitz <nnorwitz@gmail.com>
parents: 39491
diff changeset
2986 #endif
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2987
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2988 /* Map 16-bit characters to '\uxxxx' */
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2989 if (ch >= 256) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2990 *p++ = '\\';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2991 *p++ = 'u';
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2992 *p++ = hexdigit[(ch >> 12) & 0x000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2993 *p++ = hexdigit[(ch >> 8) & 0x000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2994 *p++ = hexdigit[(ch >> 4) & 0x000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
2995 *p++ = hexdigit[ch & 0x000F];
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2996 }
18828
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
2997
16226
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
2998 /* Map special whitespace to '\t', \n', '\r' */
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
2999 else if (ch == '\t') {
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3000 *p++ = '\\';
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3001 *p++ = 't';
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3002 }
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3003 else if (ch == '\n') {
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3004 *p++ = '\\';
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3005 *p++ = 'n';
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3006 }
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3007 else if (ch == '\r') {
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3008 *p++ = '\\';
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3009 *p++ = 'r';
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3010 }
18828
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
3011
16226
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3012 /* Map non-printable US ASCII to '\xhh' */
21067
c9c4c523dcbf Fix for bug #485951: repr diff between string and unicode.
Marc-André Lemburg <mal@egenix.com>
parents: 21028
diff changeset
3013 else if (ch < ' ' || ch >= 0x7F) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3014 *p++ = '\\';
16226
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3015 *p++ = 'x';
18774
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
3016 *p++ = hexdigit[(ch >> 4) & 0x000F];
57684d3cefce Make the unicode-escape and the UTF-16 codecs handle surrogates
Marc-André Lemburg <mal@egenix.com>
parents: 18773
diff changeset
3017 *p++ = hexdigit[ch & 0x000F];
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3018 }
18828
081d96fbcf43 Fix for bug #444493: u'\U00010001' segfaults with current CVS on
Marc-André Lemburg <mal@egenix.com>
parents: 18774
diff changeset
3019
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3020 /* Copy everything else as-is */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3021 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3022 *p++ = (char) ch;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3023 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3024 if (quotes)
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3025 *p++ = PyString_AS_STRING(repr)[1];
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3026
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3027 *p = '\0';
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3028 _PyString_Resize(&repr, p - PyString_AS_STRING(repr));
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3029 return repr;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3030 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3031
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3032 PyObject *PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3033 Py_ssize_t size)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3034 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3035 return unicodeescape_string(s, size, 0);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3036 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3037
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3038 PyObject *PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3039 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3040 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3041 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3042 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3043 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3044 return PyUnicode_EncodeUnicodeEscape(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3045 PyUnicode_GET_SIZE(unicode));
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3046 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3047
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3048 /* --- Raw Unicode Escape Codec ------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3049
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3050 PyObject *PyUnicode_DecodeRawUnicodeEscape(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3051 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3052 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3053 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3054 const char *starts = s;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3055 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3056 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3057 Py_ssize_t outpos;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3058 PyUnicodeObject *v;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3059 Py_UNICODE *p;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3060 const char *end;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3061 const char *bs;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3062 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3063 PyObject *exc = NULL;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3064
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3065 /* Escaped strings will always be longer than the resulting
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3066 Unicode string, so we start with size here and then reduce the
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3067 length after conversion to the true value. (But decoding error
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3068 handler might have to resize the string) */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3069 v = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3070 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3071 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3072 if (size == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3073 return (PyObject *)v;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3074 p = PyUnicode_AS_UNICODE(v);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3075 end = s + size;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3076 while (s < end) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3077 unsigned char c;
22423
b523a470a0ce Do not insert characters for unicode-escape decoders if the error mode
Martin v. Löwis <martin@v.loewis.de>
parents: 22087
diff changeset
3078 Py_UCS4 x;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3079 int i;
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3080 int count;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3081
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3082 /* Non-escape characters are interpreted as Unicode ordinals */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3083 if (*s != '\\') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3084 *p++ = (unsigned char)*s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3085 continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3086 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3087 startinpos = s-starts;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3088
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3089 /* \u-escapes are only interpreted iff the number of leading
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3090 backslashes if odd */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3091 bs = s;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3092 for (;s < end;) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3093 if (*s != '\\')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3094 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3095 *p++ = (unsigned char)*s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3096 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3097 if (((s - bs) & 1) == 0 ||
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3098 s >= end ||
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3099 (*s != 'u' && *s != 'U')) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3100 continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3101 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3102 p--;
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3103 count = *s=='u' ? 4 : 8;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3104 s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3105
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3106 /* \uXXXX with 4 hex digits, \Uxxxxxxxx with 8 */
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3107 outpos = p-PyUnicode_AS_UNICODE(v);
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3108 for (x = 0, i = 0; i < count; ++i, ++s) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3109 c = (unsigned char)*s;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3110 if (!isxdigit(c)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3111 endinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3112 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3113 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3114 "rawunicodeescape", "truncated \\uXXXX",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3115 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3116 (PyObject **)&v, &outpos, &p))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3117 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3118 goto nextByte;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3119 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3120 x = (x<<4) & ~0xF;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3121 if (c >= '0' && c <= '9')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3122 x += c - '0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3123 else if (c >= 'a' && c <= 'f')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3124 x += 10 + c - 'a';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3125 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3126 x += 10 + c - 'A';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3127 }
46028
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3128 if (x <= 0xffff)
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3129 /* UCS-2 character */
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3130 *p++ = (Py_UNICODE) x;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3131 else if (x <= 0x10ffff) {
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3132 /* UCS-4 character. Either store directly, or as
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3133 surrogate pair. */
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3134 #ifdef Py_UNICODE_WIDE
46059
c92300930772 Issue2469: Correct a typo I introduced at r61793: compilation error with UCS4 builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46028
diff changeset
3135 *p++ = (Py_UNICODE) x;
46028
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3136 #else
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3137 x -= 0x10000L;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3138 *p++ = 0xD800 + (Py_UNICODE) (x >> 10);
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3139 *p++ = 0xDC00 + (Py_UNICODE) (x & 0x03FF);
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3140 #endif
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3141 } else {
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3142 endinpos = s-starts;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3143 outpos = p-PyUnicode_AS_UNICODE(v);
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3144 if (unicode_decode_call_errorhandler(
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3145 errors, &errorHandler,
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3146 "rawunicodeescape", "\\Uxxxxxxxx out of range",
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3147 starts, size, &startinpos, &endinpos, &exc, &s,
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3148 (PyObject **)&v, &outpos, &p))
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3149 goto onError;
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3150 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3151 nextByte:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3152 ;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3153 }
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
3154 if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
3155 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3156 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3157 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3158 return (PyObject *)v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3159
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3160 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3161 Py_XDECREF(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3162 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3163 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3164 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3165 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3166
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3167 PyObject *PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3168 Py_ssize_t size)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3169 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3170 PyObject *repr;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3171 char *p;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3172 char *q;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3173
16226
c7c404b35f33 Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.
Ka-Ping Yee <ping@zesty.ca>
parents: 16200
diff changeset
3174 static const char *hexdigit = "0123456789abcdef";
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3175 #ifdef Py_UNICODE_WIDE
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3176 const Py_ssize_t expandsize = 10;
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3177 #else
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3178 const Py_ssize_t expandsize = 6;
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3179 #endif
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3180
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3181 if (size > PY_SSIZE_T_MAX / expandsize)
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3182 return PyErr_NoMemory();
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3183
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
3184 repr = PyString_FromStringAndSize(NULL, expandsize * size);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3185 if (repr == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3186 return NULL;
13966
91ddeed12fa7 Fixed a couple of instances where a 0-length string was being
Marc-André Lemburg <mal@egenix.com>
parents: 13946
diff changeset
3187 if (size == 0)
91ddeed12fa7 Fixed a couple of instances where a 0-length string was being
Marc-André Lemburg <mal@egenix.com>
parents: 13946
diff changeset
3188 return repr;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3189
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3190 p = q = PyString_AS_STRING(repr);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3191 while (size-- > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3192 Py_UNICODE ch = *s++;
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3193 #ifdef Py_UNICODE_WIDE
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3194 /* Map 32-bit characters to '\Uxxxxxxxx' */
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3195 if (ch >= 0x10000) {
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3196 *p++ = '\\';
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3197 *p++ = 'U';
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3198 *p++ = hexdigit[(ch >> 28) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3199 *p++ = hexdigit[(ch >> 24) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3200 *p++ = hexdigit[(ch >> 20) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3201 *p++ = hexdigit[(ch >> 16) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3202 *p++ = hexdigit[(ch >> 12) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3203 *p++ = hexdigit[(ch >> 8) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3204 *p++ = hexdigit[(ch >> 4) & 0xf];
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3205 *p++ = hexdigit[ch & 15];
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3206 }
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3207 else
46028
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3208 #else
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3209 /* Map UTF-16 surrogate pairs to '\U00xxxxxx' */
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3210 if (ch >= 0xD800 && ch < 0xDC00) {
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3211 Py_UNICODE ch2;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3212 Py_UCS4 ucs;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3213
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3214 ch2 = *s++;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3215 size--;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3216 if (ch2 >= 0xDC00 && ch2 <= 0xDFFF) {
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3217 ucs = (((ch & 0x03FF) << 10) | (ch2 & 0x03FF)) + 0x00010000;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3218 *p++ = '\\';
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3219 *p++ = 'U';
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3220 *p++ = hexdigit[(ucs >> 28) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3221 *p++ = hexdigit[(ucs >> 24) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3222 *p++ = hexdigit[(ucs >> 20) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3223 *p++ = hexdigit[(ucs >> 16) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3224 *p++ = hexdigit[(ucs >> 12) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3225 *p++ = hexdigit[(ucs >> 8) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3226 *p++ = hexdigit[(ucs >> 4) & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3227 *p++ = hexdigit[ucs & 0xf];
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3228 continue;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3229 }
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3230 /* Fall through: isolated surrogates are copied as-is */
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3231 s--;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3232 size++;
8710ba7fba2b #1477: ur'\U0010FFFF' raised in narrow unicode builds.
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 46026
diff changeset
3233 }
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
3234 #endif
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3235 /* Map 16-bit characters to '\uxxxx' */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3236 if (ch >= 256) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3237 *p++ = '\\';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3238 *p++ = 'u';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3239 *p++ = hexdigit[(ch >> 12) & 0xf];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3240 *p++ = hexdigit[(ch >> 8) & 0xf];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3241 *p++ = hexdigit[(ch >> 4) & 0xf];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3242 *p++ = hexdigit[ch & 15];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3243 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3244 /* Copy everything else as-is */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3245 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3246 *p++ = (char) ch;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3247 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3248 *p = '\0';
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3249 _PyString_Resize(&repr, p - q);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3250 return repr;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3251 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3252
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3253 PyObject *PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3254 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3255 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3256 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3257 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3258 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3259 return PyUnicode_EncodeRawUnicodeEscape(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3260 PyUnicode_GET_SIZE(unicode));
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3261 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3262
35485
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3263 /* --- Unicode Internal Codec ------------------------------------------- */
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3264
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3265 PyObject *_PyUnicode_DecodeUnicodeInternal(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3266 Py_ssize_t size,
35485
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3267 const char *errors)
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3268 {
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3269 const char *starts = s;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3270 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3271 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3272 Py_ssize_t outpos;
35485
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3273 PyUnicodeObject *v;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3274 Py_UNICODE *p;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3275 const char *end;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3276 const char *reason;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3277 PyObject *errorHandler = NULL;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3278 PyObject *exc = NULL;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3279
36236
ecd423718a32 Fix icc warnings: remove (sometimes) unused variable conditionally
Neal Norwitz <nnorwitz@gmail.com>
parents: 36203
diff changeset
3280 #ifdef Py_UNICODE_WIDE
ecd423718a32 Fix icc warnings: remove (sometimes) unused variable conditionally
Neal Norwitz <nnorwitz@gmail.com>
parents: 36203
diff changeset
3281 Py_UNICODE unimax = PyUnicode_GetMax();
ecd423718a32 Fix icc warnings: remove (sometimes) unused variable conditionally
Neal Norwitz <nnorwitz@gmail.com>
parents: 36203
diff changeset
3282 #endif
ecd423718a32 Fix icc warnings: remove (sometimes) unused variable conditionally
Neal Norwitz <nnorwitz@gmail.com>
parents: 36203
diff changeset
3283
39995
6ba402b60041 Forward-port of r52136,52138: a review of overflow-detecting code.
Armin Rigo <arigo@tunes.org>
parents: 39734
diff changeset
3284 /* XXX overflow detection missing */
35485
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3285 v = _PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3286 if (v == NULL)
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3287 goto onError;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3288 if (PyUnicode_GetSize((PyObject *)v) == 0)
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3289 return (PyObject *)v;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3290 p = PyUnicode_AS_UNICODE(v);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3291 end = s + size;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3292
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3293 while (s < end) {
38182
d8355cccff4a Patch #1488312, Fix memory alignment problem on SPARC in unicode. Will backport
Neal Norwitz <nnorwitz@gmail.com>
parents: 37775
diff changeset
3294 memcpy(p, s, sizeof(Py_UNICODE));
35485
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3295 /* We have to sanity check the raw data, otherwise doom looms for
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3296 some malformed UCS-4 data. */
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3297 if (
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3298 #ifdef Py_UNICODE_WIDE
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3299 *p > unimax || *p < 0 ||
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3300 #endif
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3301 end-s < Py_UNICODE_SIZE
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3302 )
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3303 {
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3304 startinpos = s - starts;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3305 if (end-s < Py_UNICODE_SIZE) {
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3306 endinpos = end-starts;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3307 reason = "truncated input";
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3308 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3309 else {
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3310 endinpos = s - starts + Py_UNICODE_SIZE;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3311 reason = "illegal code point (> 0x10FFFF)";
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3312 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3313 outpos = p - PyUnicode_AS_UNICODE(v);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3314 if (unicode_decode_call_errorhandler(
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3315 errors, &errorHandler,
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3316 "unicode_internal", reason,
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3317 starts, size, &startinpos, &endinpos, &exc, &s,
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3318 (PyObject **)&v, &outpos, &p)) {
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3319 goto onError;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3320 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3321 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3322 else {
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3323 p++;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3324 s += Py_UNICODE_SIZE;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3325 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3326 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3327
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
3328 if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
35485
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3329 goto onError;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3330 Py_XDECREF(errorHandler);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3331 Py_XDECREF(exc);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3332 return (PyObject *)v;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3333
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3334 onError:
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3335 Py_XDECREF(v);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3336 Py_XDECREF(errorHandler);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3337 Py_XDECREF(exc);
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3338 return NULL;
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3339 }
6b8c55877755 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34016
diff changeset
3340
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3341 /* --- Latin-1 Codec ------------------------------------------------------ */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3342
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3343 PyObject *PyUnicode_DecodeLatin1(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3344 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3345 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3346 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3347 PyUnicodeObject *v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3348 Py_UNICODE *p;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3349
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3350 /* Latin-1 is equivalent to the first 256 ordinals in Unicode. */
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
3351 if (size == 1) {
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3352 Py_UNICODE r = *(unsigned char*)s;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3353 return PyUnicode_FromUnicode(&r, 1);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3354 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3355
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3356 v = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3357 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3358 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3359 if (size == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3360 return (PyObject *)v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3361 p = PyUnicode_AS_UNICODE(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3362 while (size-- > 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3363 *p++ = (unsigned char)*s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3364 return (PyObject *)v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3365
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3366 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3367 Py_XDECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3368 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3369 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3370
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3371 /* create or adjust a UnicodeEncodeError */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3372 static void make_encode_exception(PyObject **exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3373 const char *encoding,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3374 const Py_UNICODE *unicode, Py_ssize_t size,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3375 Py_ssize_t startpos, Py_ssize_t endpos,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3376 const char *reason)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3377 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3378 if (*exceptionObject == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3379 *exceptionObject = PyUnicodeEncodeError_Create(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3380 encoding, unicode, size, startpos, endpos, reason);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3381 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3382 else {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3383 if (PyUnicodeEncodeError_SetStart(*exceptionObject, startpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3384 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3385 if (PyUnicodeEncodeError_SetEnd(*exceptionObject, endpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3386 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3387 if (PyUnicodeEncodeError_SetReason(*exceptionObject, reason))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3388 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3389 return;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3390 onError:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3391 Py_DECREF(*exceptionObject);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3392 *exceptionObject = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3393 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3394 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3395
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3396 /* raises a UnicodeEncodeError */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3397 static void raise_encode_exception(PyObject **exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3398 const char *encoding,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3399 const Py_UNICODE *unicode, Py_ssize_t size,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3400 Py_ssize_t startpos, Py_ssize_t endpos,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3401 const char *reason)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3402 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3403 make_encode_exception(exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3404 encoding, unicode, size, startpos, endpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3405 if (*exceptionObject != NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3406 PyCodec_StrictErrors(*exceptionObject);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3407 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3408
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3409 /* error handling callback helper:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3410 build arguments, call the callback and check the arguments,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3411 put the result into newpos and return the replacement string, which
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3412 has to be freed by the caller */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3413 static PyObject *unicode_encode_call_errorhandler(const char *errors,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3414 PyObject **errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3415 const char *encoding, const char *reason,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3416 const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3417 Py_ssize_t startpos, Py_ssize_t endpos,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3418 Py_ssize_t *newpos)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3419 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3420 static char *argparse = "O!n;encoding error handler must return (unicode, int) tuple";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3421
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3422 PyObject *restuple;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3423 PyObject *resunicode;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3424
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3425 if (*errorHandler == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3426 *errorHandler = PyCodec_LookupError(errors);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3427 if (*errorHandler == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3428 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3429 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3430
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3431 make_encode_exception(exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3432 encoding, unicode, size, startpos, endpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3433 if (*exceptionObject == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3434 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3435
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3436 restuple = PyObject_CallFunctionObjArgs(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3437 *errorHandler, *exceptionObject, NULL);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3438 if (restuple == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3439 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3440 if (!PyTuple_Check(restuple)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3441 PyErr_Format(PyExc_TypeError, &argparse[4]);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3442 Py_DECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3443 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3444 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3445 if (!PyArg_ParseTuple(restuple, argparse, &PyUnicode_Type,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3446 &resunicode, newpos)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3447 Py_DECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3448 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3449 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3450 if (*newpos<0)
27555
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
3451 *newpos = size+*newpos;
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
3452 if (*newpos<0 || *newpos>size) {
36510
0f3894bb2ada Support %zd in PyErr_Format and PyString_FromFormat.
Martin v. Löwis <martin@v.loewis.de>
parents: 36508
diff changeset
3453 PyErr_Format(PyExc_IndexError, "position %zd from error handler out of bounds", *newpos);
27555
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
3454 Py_DECREF(restuple);
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
3455 return NULL;
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
3456 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3457 Py_INCREF(resunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3458 Py_DECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3459 return resunicode;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3460 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3461
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3462 static PyObject *unicode_encode_ucs1(const Py_UNICODE *p,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3463 Py_ssize_t size,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3464 const char *errors,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3465 int limit)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3466 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3467 /* output object */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3468 PyObject *res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3469 /* pointers to the beginning and end+1 of input */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3470 const Py_UNICODE *startp = p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3471 const Py_UNICODE *endp = p + size;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3472 /* pointer to the beginning of the unencodable characters */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3473 /* const Py_UNICODE *badp = NULL; */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3474 /* pointer into the output */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3475 char *str;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3476 /* current output position */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3477 Py_ssize_t respos = 0;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3478 Py_ssize_t ressize;
37596
1b078493d538 More low-hanging fruit. Still need to re-arrange some code (or find a better
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37306
diff changeset
3479 const char *encoding = (limit == 256) ? "latin-1" : "ascii";
1b078493d538 More low-hanging fruit. Still need to re-arrange some code (or find a better
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37306
diff changeset
3480 const char *reason = (limit == 256) ? "ordinal not in range(256)" : "ordinal not in range(128)";
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3481 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3482 PyObject *exc = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3483 /* the following variable is used for caching string comparisons
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3484 * -1=not initialized, 0=unknown, 1=strict, 2=replace, 3=ignore, 4=xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3485 int known_errorHandler = -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3486
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3487 /* allocate enough for a simple encoding without
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3488 replacements, if we need more, we'll resize */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3489 res = PyString_FromStringAndSize(NULL, size);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3490 if (res == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3491 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3492 if (size == 0)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3493 return res;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3494 str = PyString_AS_STRING(res);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3495 ressize = size;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3496
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3497 while (p<endp) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3498 Py_UNICODE c = *p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3499
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3500 /* can we encode this? */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3501 if (c<limit) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3502 /* no overflow check, because we know that the space is enough */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3503 *str++ = (char)c;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3504 ++p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3505 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3506 else {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3507 Py_ssize_t unicodepos = p-startp;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3508 Py_ssize_t requiredsize;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3509 PyObject *repunicode;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3510 Py_ssize_t repsize;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3511 Py_ssize_t newpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3512 Py_ssize_t respos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3513 Py_UNICODE *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3514 /* startpos for collecting unencodable chars */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3515 const Py_UNICODE *collstart = p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3516 const Py_UNICODE *collend = p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3517 /* find all unecodable characters */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3518 while ((collend < endp) && ((*collend)>=limit))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3519 ++collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3520 /* cache callback name lookup (if not done yet, i.e. it's the first error) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3521 if (known_errorHandler==-1) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3522 if ((errors==NULL) || (!strcmp(errors, "strict")))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3523 known_errorHandler = 1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3524 else if (!strcmp(errors, "replace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3525 known_errorHandler = 2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3526 else if (!strcmp(errors, "ignore"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3527 known_errorHandler = 3;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3528 else if (!strcmp(errors, "xmlcharrefreplace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3529 known_errorHandler = 4;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3530 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3531 known_errorHandler = 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3532 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3533 switch (known_errorHandler) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3534 case 1: /* strict */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3535 raise_encode_exception(&exc, encoding, startp, size, collstart-startp, collend-startp, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3536 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3537 case 2: /* replace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3538 while (collstart++<collend)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3539 *str++ = '?'; /* fall through */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3540 case 3: /* ignore */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3541 p = collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3542 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3543 case 4: /* xmlcharrefreplace */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3544 respos = str-PyString_AS_STRING(res);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3545 /* determine replacement size (temporarily (mis)uses p) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3546 for (p = collstart, repsize = 0; p < collend; ++p) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3547 if (*p<10)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3548 repsize += 2+1+1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3549 else if (*p<100)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3550 repsize += 2+2+1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3551 else if (*p<1000)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3552 repsize += 2+3+1;
31248
353a16033f6a Fix broken xmlcharrefreplace by rev 2.204.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31238
diff changeset
3553 else if (*p<10000)
353a16033f6a Fix broken xmlcharrefreplace by rev 2.204.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31238
diff changeset
3554 repsize += 2+4+1;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
3555 #ifndef Py_UNICODE_WIDE
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
3556 else
31248
353a16033f6a Fix broken xmlcharrefreplace by rev 2.204.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31238
diff changeset
3557 repsize += 2+5+1;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
3558 #else
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3559 else if (*p<100000)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3560 repsize += 2+5+1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3561 else if (*p<1000000)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3562 repsize += 2+6+1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3563 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3564 repsize += 2+7+1;
31238
4edf200a8118 SF #859573: Reduce compiler warnings on gcc 3.2 and above.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31199
diff changeset
3565 #endif
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3566 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3567 requiredsize = respos+repsize+(endp-collend);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3568 if (requiredsize > ressize) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3569 if (requiredsize<2*ressize)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3570 requiredsize = 2*ressize;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3571 if (_PyString_Resize(&res, requiredsize))
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3572 goto onError;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3573 str = PyString_AS_STRING(res) + respos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3574 ressize = requiredsize;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3575 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3576 /* generate replacement (temporarily (mis)uses p) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3577 for (p = collstart; p < collend; ++p) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3578 str += sprintf(str, "&#%d;", (int)*p);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3579 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3580 p = collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3581 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3582 default:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3583 repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3584 encoding, reason, startp, size, &exc,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3585 collstart-startp, collend-startp, &newpos);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3586 if (repunicode == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3587 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3588 /* need more space? (at least enough for what we
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3589 have+the replacement+the rest of the string, so
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3590 we won't have to check space for encodable characters) */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3591 respos = str-PyString_AS_STRING(res);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3592 repsize = PyUnicode_GET_SIZE(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3593 requiredsize = respos+repsize+(endp-collend);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3594 if (requiredsize > ressize) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3595 if (requiredsize<2*ressize)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3596 requiredsize = 2*ressize;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3597 if (_PyString_Resize(&res, requiredsize)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3598 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3599 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3600 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3601 str = PyString_AS_STRING(res) + respos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3602 ressize = requiredsize;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3603 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3604 /* check if there is anything unencodable in the replacement
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3605 and copy it to the output */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3606 for (uni2 = PyUnicode_AS_UNICODE(repunicode);repsize-->0; ++uni2, ++str) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3607 c = *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3608 if (c >= limit) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3609 raise_encode_exception(&exc, encoding, startp, size,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3610 unicodepos, unicodepos+1, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3611 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3612 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3613 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3614 *str = (char)c;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3615 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3616 p = startp + newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3617 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3618 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3619 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3620 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3621 /* Resize if we allocated to much */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3622 respos = str-PyString_AS_STRING(res);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3623 if (respos<ressize)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3624 /* If this falls res will be NULL */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3625 _PyString_Resize(&res, respos);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3626 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3627 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3628 return res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3629
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3630 onError:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3631 Py_XDECREF(res);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3632 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3633 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3634 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3635 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3636
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3637 PyObject *PyUnicode_EncodeLatin1(const Py_UNICODE *p,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3638 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3639 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3640 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3641 return unicode_encode_ucs1(p, size, errors, 256);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3642 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3643
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3644 PyObject *PyUnicode_AsLatin1String(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3645 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3646 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3647 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3648 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3649 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3650 return PyUnicode_EncodeLatin1(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3651 PyUnicode_GET_SIZE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3652 NULL);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3653 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3654
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3655 /* --- 7-bit ASCII Codec -------------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3656
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3657 PyObject *PyUnicode_DecodeASCII(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3658 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3659 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3660 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3661 const char *starts = s;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3662 PyUnicodeObject *v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3663 Py_UNICODE *p;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3664 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3665 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3666 Py_ssize_t outpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3667 const char *e;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3668 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3669 PyObject *exc = NULL;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3670
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3671 /* ASCII is equivalent to the first 128 ordinals in Unicode. */
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3672 if (size == 1 && *(unsigned char*)s < 128) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3673 Py_UNICODE r = *(unsigned char*)s;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3674 return PyUnicode_FromUnicode(&r, 1);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
3675 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3676
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3677 v = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3678 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3679 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3680 if (size == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3681 return (PyObject *)v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3682 p = PyUnicode_AS_UNICODE(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3683 e = s + size;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3684 while (s < e) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3685 register unsigned char c = (unsigned char)*s;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3686 if (c < 128) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3687 *p++ = c;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3688 ++s;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3689 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3690 else {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3691 startinpos = s-starts;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3692 endinpos = startinpos + 1;
30439
30e495efe90a Double-fix of crash in Unicode freelist handling.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30431
diff changeset
3693 outpos = p - (Py_UNICODE *)PyUnicode_AS_UNICODE(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3694 if (unicode_decode_call_errorhandler(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3695 errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3696 "ascii", "ordinal not in range(128)",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3697 starts, size, &startinpos, &endinpos, &exc, &s,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3698 (PyObject **)&v, &outpos, &p))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3699 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3700 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3701 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3702 if (p - PyUnicode_AS_UNICODE(v) < PyString_GET_SIZE(v))
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
3703 if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
12329
3cc77848a62a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12320
diff changeset
3704 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3705 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3706 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3707 return (PyObject *)v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3708
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3709 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3710 Py_XDECREF(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3711 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3712 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3713 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3714 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3715
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3716 PyObject *PyUnicode_EncodeASCII(const Py_UNICODE *p,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3717 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3718 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3719 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3720 return unicode_encode_ucs1(p, size, errors, 128);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3721 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3722
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3723 PyObject *PyUnicode_AsASCIIString(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3724 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3725 if (!PyUnicode_Check(unicode)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3726 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3727 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3728 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3729 return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3730 PyUnicode_GET_SIZE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3731 NULL);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3732 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3733
24071
8142f8d668a0 Patch #569753: Remove support for WIN16.
Martin v. Löwis <martin@v.loewis.de>
parents: 23837
diff changeset
3734 #if defined(MS_WINDOWS) && defined(HAVE_USABLE_WCHAR_T)
12193
c3fc2b4617fc Some blank lines.
Guido van Rossum <guido@python.org>
parents: 12067
diff changeset
3735
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3736 /* --- MBCS codecs for Windows -------------------------------------------- */
12193
c3fc2b4617fc Some blank lines.
Guido van Rossum <guido@python.org>
parents: 12067
diff changeset
3737
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3738 #if SIZEOF_INT < SIZEOF_SSIZE_T
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3739 #define NEED_RETRY
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3740 #endif
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3741
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3742 /* XXX This code is limited to "true" double-byte encodings, as
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3743 a) it assumes an incomplete character consists of a single byte, and
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3744 b) IsDBCSLeadByte (probably) does not work for non-DBCS multi-byte
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3745 encodings, see IsDBCSLeadByteEx documentation. */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3746
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3747 static int is_dbcs_lead_byte(const char *s, int offset)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3748 {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3749 const char *curr = s + offset;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3750
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3751 if (IsDBCSLeadByte(*curr)) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3752 const char *prev = CharPrev(s, curr);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3753 return (prev == curr) || !IsDBCSLeadByte(*prev) || (curr - prev == 2);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3754 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3755 return 0;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3756 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3757
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3758 /*
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3759 * Decode MBCS string into unicode object. If 'final' is set, converts
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3760 * trailing lead-byte too. Returns consumed size if succeed, -1 otherwise.
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3761 */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3762 static int decode_mbcs(PyUnicodeObject **v,
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3763 const char *s, /* MBCS string */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3764 int size, /* sizeof MBCS string */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3765 int final)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3766 {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3767 Py_UNICODE *p;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3768 Py_ssize_t n = 0;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3769 int usize = 0;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3770
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3771 assert(size >= 0);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3772
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3773 /* Skip trailing lead-byte unless 'final' is set */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3774 if (!final && size >= 1 && is_dbcs_lead_byte(s, size - 1))
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3775 --size;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3776
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3777 /* First get the size of the result */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3778 if (size > 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3779 usize = MultiByteToWideChar(CP_ACP, 0, s, size, NULL, 0);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3780 if (usize == 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3781 PyErr_SetFromWindowsErrWithFilename(0, NULL);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3782 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3783 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3784 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3785
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3786 if (*v == NULL) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3787 /* Create unicode object */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3788 *v = _PyUnicode_New(usize);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3789 if (*v == NULL)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3790 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3791 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3792 else {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3793 /* Extend unicode object */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3794 n = PyUnicode_GET_SIZE(*v);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3795 if (_PyUnicode_Resize(v, n + usize) < 0)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3796 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3797 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3798
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3799 /* Do the conversion */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3800 if (size > 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3801 p = PyUnicode_AS_UNICODE(*v) + n;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3802 if (0 == MultiByteToWideChar(CP_ACP, 0, s, size, p, usize)) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3803 PyErr_SetFromWindowsErrWithFilename(0, NULL);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3804 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3805 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3806 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3807
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3808 return size;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3809 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3810
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3811 PyObject *PyUnicode_DecodeMBCSStateful(const char *s,
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3812 Py_ssize_t size,
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3813 const char *errors,
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3814 Py_ssize_t *consumed)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3815 {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3816 PyUnicodeObject *v = NULL;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3817 int done;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3818
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3819 if (consumed)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3820 *consumed = 0;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3821
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3822 #ifdef NEED_RETRY
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3823 retry:
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3824 if (size > INT_MAX)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3825 done = decode_mbcs(&v, s, INT_MAX, 0);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3826 else
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3827 #endif
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3828 done = decode_mbcs(&v, s, (int)size, !consumed);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3829
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3830 if (done < 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3831 Py_XDECREF(v);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3832 return NULL;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3833 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3834
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3835 if (consumed)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3836 *consumed += done;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3837
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3838 #ifdef NEED_RETRY
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3839 if (size > INT_MAX) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3840 s += done;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3841 size -= done;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3842 goto retry;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3843 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3844 #endif
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3845
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3846 return (PyObject *)v;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3847 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3848
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3849 PyObject *PyUnicode_DecodeMBCS(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3850 Py_ssize_t size,
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3851 const char *errors)
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3852 {
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3853 return PyUnicode_DecodeMBCSStateful(s, size, errors, NULL);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3854 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3855
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3856 /*
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3857 * Convert unicode into string object (MBCS).
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3858 * Returns 0 if succeed, -1 otherwise.
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3859 */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3860 static int encode_mbcs(PyObject **repr,
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3861 const Py_UNICODE *p, /* unicode */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3862 int size) /* size of unicode */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3863 {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3864 int mbcssize = 0;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3865 Py_ssize_t n = 0;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3866
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3867 assert(size >= 0);
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3868
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3869 /* First get the size of the result */
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3870 if (size > 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3871 mbcssize = WideCharToMultiByte(CP_ACP, 0, p, size, NULL, 0, NULL, NULL);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3872 if (mbcssize == 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3873 PyErr_SetFromWindowsErrWithFilename(0, NULL);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3874 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3875 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3876 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3877
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3878 if (*repr == NULL) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3879 /* Create string object */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3880 *repr = PyString_FromStringAndSize(NULL, mbcssize);
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3881 if (*repr == NULL)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3882 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3883 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3884 else {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3885 /* Extend string object */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3886 n = PyString_Size(*repr);
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3887 if (_PyString_Resize(repr, n + mbcssize) < 0)
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3888 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3889 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3890
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3891 /* Do the conversion */
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3892 if (size > 0) {
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
3893 char *s = PyString_AS_STRING(*repr) + n;
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3894 if (0 == WideCharToMultiByte(CP_ACP, 0, p, size, s, mbcssize, NULL, NULL)) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3895 PyErr_SetFromWindowsErrWithFilename(0, NULL);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3896 return -1;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3897 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3898 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3899
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3900 return 0;
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3901 }
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3902
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3903 PyObject *PyUnicode_EncodeMBCS(const Py_UNICODE *p,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3904 Py_ssize_t size,
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3905 const char *errors)
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3906 {
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3907 PyObject *repr = NULL;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3908 int ret;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3909
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3910 #ifdef NEED_RETRY
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3911 retry:
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3912 if (size > INT_MAX)
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3913 ret = encode_mbcs(&repr, p, INT_MAX);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3914 else
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3915 #endif
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3916 ret = encode_mbcs(&repr, p, (int)size);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3917
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3918 if (ret < 0) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3919 Py_XDECREF(repr);
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3920 return NULL;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3921 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3922
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3923 #ifdef NEED_RETRY
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3924 if (size > INT_MAX) {
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3925 p += INT_MAX;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3926 size -= INT_MAX;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3927 goto retry;
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3928 }
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3929 #endif
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3930
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3931 return repr;
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3932 }
12193
c3fc2b4617fc Some blank lines.
Guido van Rossum <guido@python.org>
parents: 12067
diff changeset
3933
29706
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3934 PyObject *PyUnicode_AsMBCSString(PyObject *unicode)
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3935 {
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3936 if (!PyUnicode_Check(unicode)) {
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3937 PyErr_BadArgument();
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3938 return NULL;
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3939 }
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3940 return PyUnicode_EncodeMBCS(PyUnicode_AS_UNICODE(unicode),
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3941 PyUnicode_GET_SIZE(unicode),
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3942 NULL);
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3943 }
a4fead0e24f8 Support 'mbcs' as a 'built-in' encoding, so the C API can use it without
Mark Hammond <mhammond@skippinet.com.au>
parents: 29641
diff changeset
3944
38817
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3945 #undef NEED_RETRY
549c547700af Patch #1455898: Incremental mode for "mbcs" codec.
Martin v. Löwis <martin@v.loewis.de>
parents: 38786
diff changeset
3946
24071
8142f8d668a0 Patch #569753: Remove support for WIN16.
Martin v. Löwis <martin@v.loewis.de>
parents: 23837
diff changeset
3947 #endif /* MS_WINDOWS */
12067
bcab189ddf86 MBCS codecs. (Win32 only.) By Mark Hammond.
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
3948
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3949 /* --- Character Mapping Codec -------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3950
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3951 PyObject *PyUnicode_DecodeCharmap(const char *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3952 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3953 PyObject *mapping,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3954 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3955 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3956 const char *starts = s;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3957 Py_ssize_t startinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3958 Py_ssize_t endinpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3959 Py_ssize_t outpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3960 const char *e;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3961 PyUnicodeObject *v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3962 Py_UNICODE *p;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3963 Py_ssize_t extrachars = 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3964 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3965 PyObject *exc = NULL;
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3966 Py_UNICODE *mapstring = NULL;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
3967 Py_ssize_t maplen = 0;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
3968
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3969 /* Default to Latin-1 */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3970 if (mapping == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3971 return PyUnicode_DecodeLatin1(s, size, errors);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3972
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3973 v = _PyUnicode_New(size);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3974 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3975 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3976 if (size == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3977 return (PyObject *)v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3978 p = PyUnicode_AS_UNICODE(v);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
3979 e = s + size;
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3980 if (PyUnicode_CheckExact(mapping)) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3981 mapstring = PyUnicode_AS_UNICODE(mapping);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3982 maplen = PyUnicode_GET_SIZE(mapping);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3983 while (s < e) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3984 unsigned char ch = *s;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3985 Py_UNICODE x = 0xfffe; /* illegal value */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3986
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3987 if (ch < maplen)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3988 x = mapstring[ch];
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3989
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3990 if (x == 0xfffe) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3991 /* undefined mapping */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3992 outpos = p-PyUnicode_AS_UNICODE(v);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3993 startinpos = s-starts;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3994 endinpos = startinpos+1;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3995 if (unicode_decode_call_errorhandler(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3996 errors, &errorHandler,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3997 "charmap", "character maps to <undefined>",
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3998 starts, size, &startinpos, &endinpos, &exc, &s,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
3999 (PyObject **)&v, &outpos, &p)) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4000 goto onError;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4001 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4002 continue;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4003 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4004 *p++ = x;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4005 ++s;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4006 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4007 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4008 else {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4009 while (s < e) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4010 unsigned char ch = *s;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4011 PyObject *w, *x;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4012
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4013 /* Get mapping (char ordinal -> integer, Unicode char or None) */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4014 w = PyInt_FromLong((long)ch);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4015 if (w == NULL)
15910
f7f3be3d00b9 Added checks to prevent PyUnicode_Count() from dumping core
Marc-André Lemburg <mal@egenix.com>
parents: 15749
diff changeset
4016 goto onError;
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4017 x = PyObject_GetItem(mapping, w);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4018 Py_DECREF(w);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4019 if (x == NULL) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4020 if (PyErr_ExceptionMatches(PyExc_LookupError)) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4021 /* No mapping found means: mapping is undefined. */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4022 PyErr_Clear();
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4023 x = Py_None;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4024 Py_INCREF(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4025 } else
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4026 goto onError;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4027 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4028
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4029 /* Apply mapping */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4030 if (PyInt_Check(x)) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4031 long value = PyInt_AS_LONG(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4032 if (value < 0 || value > 65535) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4033 PyErr_SetString(PyExc_TypeError,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4034 "character mapping must be in range(65536)");
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4035 Py_DECREF(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4036 goto onError;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4037 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4038 *p++ = (Py_UNICODE)value;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4039 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4040 else if (x == Py_None) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4041 /* undefined mapping */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4042 outpos = p-PyUnicode_AS_UNICODE(v);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4043 startinpos = s-starts;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4044 endinpos = startinpos+1;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4045 if (unicode_decode_call_errorhandler(
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4046 errors, &errorHandler,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4047 "charmap", "character maps to <undefined>",
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4048 starts, size, &startinpos, &endinpos, &exc, &s,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4049 (PyObject **)&v, &outpos, &p)) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4050 Py_DECREF(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4051 goto onError;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4052 }
35962
73ebdc62bf8f Fix leaked reference to None.
Walter Dörwald <walter@livinglogic.de>
parents: 35827
diff changeset
4053 Py_DECREF(x);
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4054 continue;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4055 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4056 else if (PyUnicode_Check(x)) {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4057 Py_ssize_t targetsize = PyUnicode_GET_SIZE(x);
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4058
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4059 if (targetsize == 1)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4060 /* 1-1 mapping */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4061 *p++ = *PyUnicode_AS_UNICODE(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4062
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4063 else if (targetsize > 1) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4064 /* 1-n mapping */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4065 if (targetsize > extrachars) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4066 /* resize first */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4067 Py_ssize_t oldpos = p - PyUnicode_AS_UNICODE(v);
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4068 Py_ssize_t needed = (targetsize - extrachars) + \
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4069 (targetsize << 2);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4070 extrachars += needed;
39995
6ba402b60041 Forward-port of r52136,52138: a review of overflow-detecting code.
Armin Rigo <arigo@tunes.org>
parents: 39734
diff changeset
4071 /* XXX overflow detection missing */
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4072 if (_PyUnicode_Resize(&v,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4073 PyUnicode_GET_SIZE(v) + needed) < 0) {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4074 Py_DECREF(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4075 goto onError;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4076 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4077 p = PyUnicode_AS_UNICODE(v) + oldpos;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4078 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4079 Py_UNICODE_COPY(p,
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4080 PyUnicode_AS_UNICODE(x),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4081 targetsize);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4082 p += targetsize;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4083 extrachars -= targetsize;
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4084 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4085 /* 1-0 mapping: skip the character */
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4086 }
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4087 else {
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4088 /* wrong return value */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4089 PyErr_SetString(PyExc_TypeError,
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4090 "character mapping must return integer, None or unicode");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4091 Py_DECREF(x);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4092 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4093 }
35702
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4094 Py_DECREF(x);
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending
Walter Dörwald <walter@livinglogic.de>
parents: 35485
diff changeset
4095 ++s;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4096 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4097 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4098 if (p - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v))
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
4099 if (_PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4100 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4101 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4102 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4103 return (PyObject *)v;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
4104
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4105 onError:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4106 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4107 Py_XDECREF(exc);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4108 Py_XDECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4109 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4110 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4111
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4112 /* Charmap encoding: the lookup table */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4113
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4114 struct encoding_map{
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4115 PyObject_HEAD
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4116 unsigned char level1[32];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4117 int count2, count3;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4118 unsigned char level23[1];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4119 };
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4120
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4121 static PyObject*
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4122 encoding_map_size(PyObject *obj, PyObject* args)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4123 {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4124 struct encoding_map *map = (struct encoding_map*)obj;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4125 return PyInt_FromLong(sizeof(*map) - 1 + 16*map->count2 +
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4126 128*map->count3);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4127 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4128
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4129 static PyMethodDef encoding_map_methods[] = {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4130 {"size", encoding_map_size, METH_NOARGS,
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4131 PyDoc_STR("Return the size (in bytes) of this object") },
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4132 { 0 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4133 };
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4134
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4135 static void
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4136 encoding_map_dealloc(PyObject* o)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4137 {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4138 PyObject_FREE(o);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4139 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4140
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4141 static PyTypeObject EncodingMapType = {
42333
6620a510e9a9 PEP 3123: Provide forward compatibility with Python 3.0, while keeping
Martin v. Löwis <martin@v.loewis.de>
parents: 42216
diff changeset
4142 PyVarObject_HEAD_INIT(NULL, 0)
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4143 "EncodingMap", /*tp_name*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4144 sizeof(struct encoding_map), /*tp_basicsize*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4145 0, /*tp_itemsize*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4146 /* methods */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4147 encoding_map_dealloc, /*tp_dealloc*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4148 0, /*tp_print*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4149 0, /*tp_getattr*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4150 0, /*tp_setattr*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4151 0, /*tp_compare*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4152 0, /*tp_repr*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4153 0, /*tp_as_number*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4154 0, /*tp_as_sequence*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4155 0, /*tp_as_mapping*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4156 0, /*tp_hash*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4157 0, /*tp_call*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4158 0, /*tp_str*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4159 0, /*tp_getattro*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4160 0, /*tp_setattro*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4161 0, /*tp_as_buffer*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4162 Py_TPFLAGS_DEFAULT, /*tp_flags*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4163 0, /*tp_doc*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4164 0, /*tp_traverse*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4165 0, /*tp_clear*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4166 0, /*tp_richcompare*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4167 0, /*tp_weaklistoffset*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4168 0, /*tp_iter*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4169 0, /*tp_iternext*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4170 encoding_map_methods, /*tp_methods*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4171 0, /*tp_members*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4172 0, /*tp_getset*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4173 0, /*tp_base*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4174 0, /*tp_dict*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4175 0, /*tp_descr_get*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4176 0, /*tp_descr_set*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4177 0, /*tp_dictoffset*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4178 0, /*tp_init*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4179 0, /*tp_alloc*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4180 0, /*tp_new*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4181 0, /*tp_free*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4182 0, /*tp_is_gc*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4183 };
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4184
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4185 PyObject*
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4186 PyUnicode_BuildEncodingMap(PyObject* string)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4187 {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4188 Py_UNICODE *decode;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4189 PyObject *result;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4190 struct encoding_map *mresult;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4191 int i;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4192 int need_dict = 0;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4193 unsigned char level1[32];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4194 unsigned char level2[512];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4195 unsigned char *mlevel1, *mlevel2, *mlevel3;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4196 int count2 = 0, count3 = 0;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4197
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4198 if (!PyUnicode_Check(string) || PyUnicode_GetSize(string) != 256) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4199 PyErr_BadArgument();
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4200 return NULL;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4201 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4202 decode = PyUnicode_AS_UNICODE(string);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4203 memset(level1, 0xFF, sizeof level1);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4204 memset(level2, 0xFF, sizeof level2);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4205
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4206 /* If there isn't a one-to-one mapping of NULL to \0,
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4207 or if there are non-BMP characters, we need to use
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4208 a mapping dictionary. */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4209 if (decode[0] != 0)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4210 need_dict = 1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4211 for (i = 1; i < 256; i++) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4212 int l1, l2;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4213 if (decode[i] == 0
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4214 #ifdef Py_UNICODE_WIDE
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4215 || decode[i] > 0xFFFF
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4216 #endif
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4217 ) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4218 need_dict = 1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4219 break;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4220 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4221 if (decode[i] == 0xFFFE)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4222 /* unmapped character */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4223 continue;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4224 l1 = decode[i] >> 11;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4225 l2 = decode[i] >> 7;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4226 if (level1[l1] == 0xFF)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4227 level1[l1] = count2++;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4228 if (level2[l2] == 0xFF)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4229 level2[l2] = count3++;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4230 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4231
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4232 if (count2 >= 0xFF || count3 >= 0xFF)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4233 need_dict = 1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4234
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4235 if (need_dict) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4236 PyObject *result = PyDict_New();
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4237 PyObject *key, *value;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4238 if (!result)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4239 return NULL;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4240 for (i = 0; i < 256; i++) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4241 key = value = NULL;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4242 key = PyInt_FromLong(decode[i]);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4243 value = PyInt_FromLong(i);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4244 if (!key || !value)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4245 goto failed1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4246 if (PyDict_SetItem(result, key, value) == -1)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4247 goto failed1;
38587
21969b71c36d Repair refleaks in unicodeobject.
Georg Brandl <georg@python.org>
parents: 38586
diff changeset
4248 Py_DECREF(key);
21969b71c36d Repair refleaks in unicodeobject.
Georg Brandl <georg@python.org>
parents: 38586
diff changeset
4249 Py_DECREF(value);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4250 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4251 return result;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4252 failed1:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4253 Py_XDECREF(key);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4254 Py_XDECREF(value);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4255 Py_DECREF(result);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4256 return NULL;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4257 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4258
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4259 /* Create a three-level trie */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4260 result = PyObject_MALLOC(sizeof(struct encoding_map) +
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4261 16*count2 + 128*count3 - 1);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4262 if (!result)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4263 return PyErr_NoMemory();
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4264 PyObject_Init(result, &EncodingMapType);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4265 mresult = (struct encoding_map*)result;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4266 mresult->count2 = count2;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4267 mresult->count3 = count3;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4268 mlevel1 = mresult->level1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4269 mlevel2 = mresult->level23;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4270 mlevel3 = mresult->level23 + 16*count2;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4271 memcpy(mlevel1, level1, 32);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4272 memset(mlevel2, 0xFF, 16*count2);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4273 memset(mlevel3, 0, 128*count3);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4274 count3 = 0;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4275 for (i = 1; i < 256; i++) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4276 int o1, o2, o3, i2, i3;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4277 if (decode[i] == 0xFFFE)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4278 /* unmapped character */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4279 continue;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4280 o1 = decode[i]>>11;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4281 o2 = (decode[i]>>7) & 0xF;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4282 i2 = 16*mlevel1[o1] + o2;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4283 if (mlevel2[i2] == 0xFF)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4284 mlevel2[i2] = count3++;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4285 o3 = decode[i] & 0x7F;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4286 i3 = 128*mlevel2[i2] + o3;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4287 mlevel3[i3] = i;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4288 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4289 return result;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4290 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4291
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4292 static int
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4293 encoding_map_lookup(Py_UNICODE c, PyObject *mapping)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4294 {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4295 struct encoding_map *map = (struct encoding_map*)mapping;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4296 int l1 = c>>11;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4297 int l2 = (c>>7) & 0xF;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4298 int l3 = c & 0x7F;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4299 int i;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4300
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4301 #ifdef Py_UNICODE_WIDE
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4302 if (c > 0xFFFF) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4303 return -1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4304 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4305 #endif
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4306 if (c == 0)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4307 return 0;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4308 /* level 1*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4309 i = map->level1[l1];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4310 if (i == 0xFF) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4311 return -1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4312 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4313 /* level 2*/
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4314 i = map->level23[16*i+l2];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4315 if (i == 0xFF) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4316 return -1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4317 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4318 /* level 3 */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4319 i = map->level23[16*map->count2 + 128*i + l3];
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4320 if (i == 0) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4321 return -1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4322 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4323 return i;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4324 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4325
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4326 /* Lookup the character ch in the mapping. If the character
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4327 can't be found, Py_None is returned (or NULL, if another
35806
44fe157255bc fix typos, mostly in comments
Fred Drake <fdrake@acm.org>
parents: 35758
diff changeset
4328 error occurred). */
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4329 static PyObject *charmapencode_lookup(Py_UNICODE c, PyObject *mapping)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4330 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4331 PyObject *w = PyInt_FromLong((long)c);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4332 PyObject *x;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4333
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4334 if (w == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4335 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4336 x = PyObject_GetItem(mapping, w);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4337 Py_DECREF(w);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4338 if (x == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4339 if (PyErr_ExceptionMatches(PyExc_LookupError)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4340 /* No mapping found means: mapping is undefined. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4341 PyErr_Clear();
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4342 x = Py_None;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4343 Py_INCREF(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4344 return x;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4345 } else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4346 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4347 }
27175
47b134bb51c3 Fix charmapencode_lookup(), so that a None value in the mapping
Walter Dörwald <walter@livinglogic.de>
parents: 27170
diff changeset
4348 else if (x == Py_None)
47b134bb51c3 Fix charmapencode_lookup(), so that a None value in the mapping
Walter Dörwald <walter@livinglogic.de>
parents: 27170
diff changeset
4349 return x;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4350 else if (PyInt_Check(x)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4351 long value = PyInt_AS_LONG(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4352 if (value < 0 || value > 255) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4353 PyErr_SetString(PyExc_TypeError,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4354 "character mapping must be in range(256)");
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4355 Py_DECREF(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4356 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4357 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4358 return x;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4359 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4360 else if (PyString_Check(x))
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4361 return x;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4362 else {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4363 /* wrong return value */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4364 PyErr_SetString(PyExc_TypeError,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4365 "character mapping must return integer, None or str");
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4366 Py_DECREF(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4367 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4368 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4369 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4370
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4371 static int
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4372 charmapencode_resize(PyObject **outobj, Py_ssize_t *outpos, Py_ssize_t requiredsize)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4373 {
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4374 Py_ssize_t outsize = PyString_GET_SIZE(*outobj);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4375 /* exponentially overallocate to minimize reallocations */
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4376 if (requiredsize < 2*outsize)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4377 requiredsize = 2*outsize;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4378 if (_PyString_Resize(outobj, requiredsize)) {
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4379 return 0;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4380 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4381 return 1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4382 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4383
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4384 typedef enum charmapencode_result {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4385 enc_SUCCESS, enc_FAILED, enc_EXCEPTION
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4386 }charmapencode_result;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4387 /* lookup the character, put the result in the output string and adjust
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4388 various state variables. Reallocate the output string if not enough
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4389 space is available. Return a new reference to the object that
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4390 was put in the output buffer, or Py_None, if the mapping was undefined
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4391 (in which case no character was written) or NULL, if a
35827
ab3d1e6a0237 Another comment typo fix
Andrew M. Kuchling <amk@amk.ca>
parents: 35826
diff changeset
4392 reallocation error occurred. The caller must decref the result */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4393 static
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4394 charmapencode_result charmapencode_output(Py_UNICODE c, PyObject *mapping,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4395 PyObject **outobj, Py_ssize_t *outpos)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4396 {
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4397 PyObject *rep;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4398 char *outstart;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4399 Py_ssize_t outsize = PyString_GET_SIZE(*outobj);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4400
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
4401 if (Py_TYPE(mapping) == &EncodingMapType) {
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4402 int res = encoding_map_lookup(c, mapping);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4403 Py_ssize_t requiredsize = *outpos+1;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4404 if (res == -1)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4405 return enc_FAILED;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4406 if (outsize<requiredsize)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4407 if (!charmapencode_resize(outobj, outpos, requiredsize))
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4408 return enc_EXCEPTION;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4409 outstart = PyString_AS_STRING(*outobj);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4410 outstart[(*outpos)++] = (char)res;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4411 return enc_SUCCESS;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4412 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4413
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4414 rep = charmapencode_lookup(c, mapping);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4415 if (rep==NULL)
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4416 return enc_EXCEPTION;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4417 else if (rep==Py_None) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4418 Py_DECREF(rep);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4419 return enc_FAILED;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4420 } else {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4421 if (PyInt_Check(rep)) {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4422 Py_ssize_t requiredsize = *outpos+1;
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4423 if (outsize<requiredsize)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4424 if (!charmapencode_resize(outobj, outpos, requiredsize)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4425 Py_DECREF(rep);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4426 return enc_EXCEPTION;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4427 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4428 outstart = PyString_AS_STRING(*outobj);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4429 outstart[(*outpos)++] = (char)PyInt_AS_LONG(rep);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4430 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4431 else {
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4432 const char *repchars = PyString_AS_STRING(rep);
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4433 Py_ssize_t repsize = PyString_GET_SIZE(rep);
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4434 Py_ssize_t requiredsize = *outpos+repsize;
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4435 if (outsize<requiredsize)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4436 if (!charmapencode_resize(outobj, outpos, requiredsize)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4437 Py_DECREF(rep);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4438 return enc_EXCEPTION;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4439 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4440 outstart = PyString_AS_STRING(*outobj);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4441 memcpy(outstart + *outpos, repchars, repsize);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4442 *outpos += repsize;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4443 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4444 }
38587
21969b71c36d Repair refleaks in unicodeobject.
Georg Brandl <georg@python.org>
parents: 38586
diff changeset
4445 Py_DECREF(rep);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4446 return enc_SUCCESS;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4447 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4448
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4449 /* handle an error in PyUnicode_EncodeCharmap
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4450 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4451 static
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4452 int charmap_encoding_error(
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4453 const Py_UNICODE *p, Py_ssize_t size, Py_ssize_t *inpos, PyObject *mapping,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4454 PyObject **exceptionObject,
30149
59bad7efff02 Fix refcount leak in PyUnicode_EncodeCharmap(). The bug surfaces
Walter Dörwald <walter@livinglogic.de>
parents: 29706
diff changeset
4455 int *known_errorHandler, PyObject **errorHandler, const char *errors,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4456 PyObject **res, Py_ssize_t *respos)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4457 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4458 PyObject *repunicode = NULL; /* initialize to prevent gcc warning */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4459 Py_ssize_t repsize;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4460 Py_ssize_t newpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4461 Py_UNICODE *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4462 /* startpos for collecting unencodable chars */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4463 Py_ssize_t collstartpos = *inpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4464 Py_ssize_t collendpos = *inpos+1;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4465 Py_ssize_t collpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4466 char *encoding = "charmap";
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4467 char *reason = "character maps to <undefined>";
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4468 charmapencode_result x;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4469
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4470 /* find all unencodable characters */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4471 while (collendpos < size) {
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4472 PyObject *rep;
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
4473 if (Py_TYPE(mapping) == &EncodingMapType) {
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4474 int res = encoding_map_lookup(p[collendpos], mapping);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4475 if (res != -1)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4476 break;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4477 ++collendpos;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4478 continue;
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4479 }
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4480
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4481 rep = charmapencode_lookup(p[collendpos], mapping);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4482 if (rep==NULL)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4483 return -1;
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4484 else if (rep!=Py_None) {
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4485 Py_DECREF(rep);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4486 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4487 }
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4488 Py_DECREF(rep);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4489 ++collendpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4490 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4491 /* cache callback name lookup
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4492 * (if not done yet, i.e. it's the first error) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4493 if (*known_errorHandler==-1) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4494 if ((errors==NULL) || (!strcmp(errors, "strict")))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4495 *known_errorHandler = 1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4496 else if (!strcmp(errors, "replace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4497 *known_errorHandler = 2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4498 else if (!strcmp(errors, "ignore"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4499 *known_errorHandler = 3;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4500 else if (!strcmp(errors, "xmlcharrefreplace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4501 *known_errorHandler = 4;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4502 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4503 *known_errorHandler = 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4504 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4505 switch (*known_errorHandler) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4506 case 1: /* strict */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4507 raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4508 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4509 case 2: /* replace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4510 for (collpos = collstartpos; collpos<collendpos; ++collpos) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4511 x = charmapencode_output('?', mapping, res, respos);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4512 if (x==enc_EXCEPTION) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4513 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4514 }
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4515 else if (x==enc_FAILED) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4516 raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4517 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4518 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4519 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4520 /* fall through */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4521 case 3: /* ignore */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4522 *inpos = collendpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4523 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4524 case 4: /* xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4525 /* generate replacement (temporarily (mis)uses p) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4526 for (collpos = collstartpos; collpos < collendpos; ++collpos) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4527 char buffer[2+29+1+1];
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4528 char *cp;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4529 sprintf(buffer, "&#%d;", (int)p[collpos]);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4530 for (cp = buffer; *cp; ++cp) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4531 x = charmapencode_output(*cp, mapping, res, respos);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4532 if (x==enc_EXCEPTION)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4533 return -1;
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4534 else if (x==enc_FAILED) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4535 raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4536 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4537 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4538 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4539 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4540 *inpos = collendpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4541 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4542 default:
30149
59bad7efff02 Fix refcount leak in PyUnicode_EncodeCharmap(). The bug surfaces
Walter Dörwald <walter@livinglogic.de>
parents: 29706
diff changeset
4543 repunicode = unicode_encode_call_errorhandler(errors, errorHandler,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4544 encoding, reason, p, size, exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4545 collstartpos, collendpos, &newpos);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4546 if (repunicode == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4547 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4548 /* generate replacement */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4549 repsize = PyUnicode_GET_SIZE(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4550 for (uni2 = PyUnicode_AS_UNICODE(repunicode); repsize-->0; ++uni2) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4551 x = charmapencode_output(*uni2, mapping, res, respos);
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4552 if (x==enc_EXCEPTION) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4553 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4554 }
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4555 else if (x==enc_FAILED) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4556 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4557 raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4558 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4559 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4560 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4561 *inpos = newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4562 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4563 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4564 return 0;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4565 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4566
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4567 PyObject *PyUnicode_EncodeCharmap(const Py_UNICODE *p,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4568 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4569 PyObject *mapping,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4570 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4571 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4572 /* output object */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4573 PyObject *res = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4574 /* current input position */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4575 Py_ssize_t inpos = 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4576 /* current output position */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4577 Py_ssize_t respos = 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4578 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4579 PyObject *exc = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4580 /* the following variable is used for caching string comparisons
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4581 * -1=not initialized, 0=unknown, 1=strict, 2=replace,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4582 * 3=ignore, 4=xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4583 int known_errorHandler = -1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4584
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4585 /* Default to Latin-1 */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4586 if (mapping == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4587 return PyUnicode_EncodeLatin1(p, size, errors);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4588
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4589 /* allocate enough for a simple encoding without
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4590 replacements, if we need more, we'll resize */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4591 res = PyString_FromStringAndSize(NULL, size);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4592 if (res == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4593 goto onError;
13966
91ddeed12fa7 Fixed a couple of instances where a 0-length string was being
Marc-André Lemburg <mal@egenix.com>
parents: 13946
diff changeset
4594 if (size == 0)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4595 return res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4596
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4597 while (inpos<size) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4598 /* try to encode it */
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4599 charmapencode_result x = charmapencode_output(p[inpos], mapping, &res, &respos);
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4600 if (x==enc_EXCEPTION) /* error */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4601 goto onError;
38586
9e0b050ce46f Patch #1359618: Speed-up charmap encoder.
Martin v. Löwis <martin@v.loewis.de>
parents: 38441
diff changeset
4602 if (x==enc_FAILED) { /* unencodable character */
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4603 if (charmap_encoding_error(p, size, &inpos, mapping,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4604 &exc,
30149
59bad7efff02 Fix refcount leak in PyUnicode_EncodeCharmap(). The bug surfaces
Walter Dörwald <walter@livinglogic.de>
parents: 29706
diff changeset
4605 &known_errorHandler, &errorHandler, errors,
30162
443533d08783 Fix another refcounting leak in PyUnicode_EncodeCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30160
diff changeset
4606 &res, &respos)) {
15910
f7f3be3d00b9 Added checks to prevent PyUnicode_Count() from dumping core
Marc-André Lemburg <mal@egenix.com>
parents: 15749
diff changeset
4607 goto onError;
30162
443533d08783 Fix another refcounting leak in PyUnicode_EncodeCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30160
diff changeset
4608 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4609 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4610 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4611 /* done with this character => adjust input position */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4612 ++inpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4613 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4614
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4615 /* Resize if we allocated to much */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4616 if (respos<PyString_GET_SIZE(res)) {
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
4617 if (_PyString_Resize(&res, respos))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4618 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4619 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4620 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4621 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4622 return res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4623
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4624 onError:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4625 Py_XDECREF(res);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4626 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4627 Py_XDECREF(errorHandler);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4628 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4629 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4630
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4631 PyObject *PyUnicode_AsCharmapString(PyObject *unicode,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4632 PyObject *mapping)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4633 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4634 if (!PyUnicode_Check(unicode) || mapping == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4635 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4636 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4637 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4638 return PyUnicode_EncodeCharmap(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4639 PyUnicode_GET_SIZE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4640 mapping,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4641 NULL);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4642 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4643
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4644 /* create or adjust a UnicodeTranslateError */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4645 static void make_translate_exception(PyObject **exceptionObject,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4646 const Py_UNICODE *unicode, Py_ssize_t size,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4647 Py_ssize_t startpos, Py_ssize_t endpos,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4648 const char *reason)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4649 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4650 if (*exceptionObject == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4651 *exceptionObject = PyUnicodeTranslateError_Create(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4652 unicode, size, startpos, endpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4653 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4654 else {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4655 if (PyUnicodeTranslateError_SetStart(*exceptionObject, startpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4656 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4657 if (PyUnicodeTranslateError_SetEnd(*exceptionObject, endpos))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4658 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4659 if (PyUnicodeTranslateError_SetReason(*exceptionObject, reason))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4660 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4661 return;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4662 onError:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4663 Py_DECREF(*exceptionObject);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4664 *exceptionObject = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4665 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4666 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4667
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4668 /* raises a UnicodeTranslateError */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4669 static void raise_translate_exception(PyObject **exceptionObject,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4670 const Py_UNICODE *unicode, Py_ssize_t size,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4671 Py_ssize_t startpos, Py_ssize_t endpos,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4672 const char *reason)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4673 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4674 make_translate_exception(exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4675 unicode, size, startpos, endpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4676 if (*exceptionObject != NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4677 PyCodec_StrictErrors(*exceptionObject);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4678 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4679
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4680 /* error handling callback helper:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4681 build arguments, call the callback and check the arguments,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4682 put the result into newpos and return the replacement string, which
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4683 has to be freed by the caller */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4684 static PyObject *unicode_translate_call_errorhandler(const char *errors,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4685 PyObject **errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4686 const char *reason,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4687 const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4688 Py_ssize_t startpos, Py_ssize_t endpos,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4689 Py_ssize_t *newpos)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4690 {
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
4691 static char *argparse = "O!n;translating error handler must return (unicode, int) tuple";
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
4692
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
4693 Py_ssize_t i_newpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4694 PyObject *restuple;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4695 PyObject *resunicode;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4696
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4697 if (*errorHandler == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4698 *errorHandler = PyCodec_LookupError(errors);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4699 if (*errorHandler == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4700 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4701 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4702
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4703 make_translate_exception(exceptionObject,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4704 unicode, size, startpos, endpos, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4705 if (*exceptionObject == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4706 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4707
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4708 restuple = PyObject_CallFunctionObjArgs(
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4709 *errorHandler, *exceptionObject, NULL);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4710 if (restuple == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4711 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4712 if (!PyTuple_Check(restuple)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4713 PyErr_Format(PyExc_TypeError, &argparse[4]);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4714 Py_DECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4715 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4716 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4717 if (!PyArg_ParseTuple(restuple, argparse, &PyUnicode_Type,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4718 &resunicode, &i_newpos)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4719 Py_DECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4720 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4721 }
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4722 if (i_newpos<0)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4723 *newpos = size+i_newpos;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4724 else
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4725 *newpos = i_newpos;
27555
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
4726 if (*newpos<0 || *newpos>size) {
36510
0f3894bb2ada Support %zd in PyErr_Format and PyString_FromFormat.
Martin v. Löwis <martin@v.loewis.de>
parents: 36508
diff changeset
4727 PyErr_Format(PyExc_IndexError, "position %zd from error handler out of bounds", *newpos);
27555
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
4728 Py_DECREF(restuple);
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
4729 return NULL;
c267f28724dd Change the treatment of positions returned by PEP293
Walter Dörwald <walter@livinglogic.de>
parents: 27521
diff changeset
4730 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4731 Py_INCREF(resunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4732 Py_DECREF(restuple);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4733 return resunicode;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4734 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4735
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4736 /* Lookup the character ch in the mapping and put the result in result,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4737 which must be decrefed by the caller.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4738 Return 0 on success, -1 on error */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4739 static
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4740 int charmaptranslate_lookup(Py_UNICODE c, PyObject *mapping, PyObject **result)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4741 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4742 PyObject *w = PyInt_FromLong((long)c);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4743 PyObject *x;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4744
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4745 if (w == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4746 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4747 x = PyObject_GetItem(mapping, w);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4748 Py_DECREF(w);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4749 if (x == NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4750 if (PyErr_ExceptionMatches(PyExc_LookupError)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4751 /* No mapping found means: use 1:1 mapping. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4752 PyErr_Clear();
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4753 *result = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4754 return 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4755 } else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4756 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4757 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4758 else if (x == Py_None) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4759 *result = x;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4760 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4761 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4762 else if (PyInt_Check(x)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4763 long value = PyInt_AS_LONG(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4764 long max = PyUnicode_GetMax();
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4765 if (value < 0 || value > max) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4766 PyErr_Format(PyExc_TypeError,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4767 "character mapping must be in range(0x%lx)", max+1);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4768 Py_DECREF(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4769 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4770 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4771 *result = x;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4772 return 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4773 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4774 else if (PyUnicode_Check(x)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4775 *result = x;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4776 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4777 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4778 else {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4779 /* wrong return value */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4780 PyErr_SetString(PyExc_TypeError,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4781 "character mapping must return integer, None or unicode");
30164
7cb4ad5932e6 Fix refcounting leak in charmaptranslate_lookup()
Walter Dörwald <walter@livinglogic.de>
parents: 30162
diff changeset
4782 Py_DECREF(x);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4783 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4784 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4785 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4786 /* ensure that *outobj is at least requiredsize characters long,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4787 if not reallocate and adjust various state variables.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4788 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4789 static
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4790 int charmaptranslate_makespace(PyObject **outobj, Py_UNICODE **outp,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4791 Py_ssize_t requiredsize)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4792 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4793 Py_ssize_t oldsize = PyUnicode_GET_SIZE(*outobj);
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4794 if (requiredsize > oldsize) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4795 /* remember old output position */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4796 Py_ssize_t outpos = *outp-PyUnicode_AS_UNICODE(*outobj);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4797 /* exponentially overallocate to minimize reallocations */
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4798 if (requiredsize < 2 * oldsize)
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4799 requiredsize = 2 * oldsize;
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
4800 if (_PyUnicode_Resize(outobj, requiredsize) < 0)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4801 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4802 *outp = PyUnicode_AS_UNICODE(*outobj) + outpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4803 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4804 return 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4805 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4806 /* lookup the character, put the result in the output string and adjust
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4807 various state variables. Return a new reference to the object that
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4808 was put in the output buffer in *result, or Py_None, if the mapping was
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4809 undefined (in which case no character was written).
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4810 The called must decref result.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4811 Return 0 on success, -1 on error. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4812 static
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4813 int charmaptranslate_output(const Py_UNICODE *startinp, const Py_UNICODE *curinp,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4814 Py_ssize_t insize, PyObject *mapping, PyObject **outobj, Py_UNICODE **outp,
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4815 PyObject **res)
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4816 {
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4817 if (charmaptranslate_lookup(*curinp, mapping, res))
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4818 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4819 if (*res==NULL) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4820 /* not found => default to 1:1 mapping */
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4821 *(*outp)++ = *curinp;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4822 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4823 else if (*res==Py_None)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4824 ;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4825 else if (PyInt_Check(*res)) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4826 /* no overflow check, because we know that the space is enough */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4827 *(*outp)++ = (Py_UNICODE)PyInt_AS_LONG(*res);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4828 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4829 else if (PyUnicode_Check(*res)) {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4830 Py_ssize_t repsize = PyUnicode_GET_SIZE(*res);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4831 if (repsize==1) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4832 /* no overflow check, because we know that the space is enough */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4833 *(*outp)++ = *PyUnicode_AS_UNICODE(*res);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4834 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4835 else if (repsize!=0) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4836 /* more than one character */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4837 Py_ssize_t requiredsize = (*outp-PyUnicode_AS_UNICODE(*outobj)) +
31455
37d501b9e2d5 Fix reallocation bug in unicode.translate(): The code was comparing
Walter Dörwald <walter@livinglogic.de>
parents: 31325
diff changeset
4838 (insize - (curinp-startinp)) +
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4839 repsize - 1;
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4840 if (charmaptranslate_makespace(outobj, outp, requiredsize))
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4841 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4842 memcpy(*outp, PyUnicode_AS_UNICODE(*res), sizeof(Py_UNICODE)*repsize);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4843 *outp += repsize;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4844 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4845 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4846 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4847 return -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4848 return 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4849 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4850
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4851 PyObject *PyUnicode_TranslateCharmap(const Py_UNICODE *p,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4852 Py_ssize_t size,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4853 PyObject *mapping,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4854 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4855 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4856 /* output object */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4857 PyObject *res = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4858 /* pointers to the beginning and end+1 of input */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4859 const Py_UNICODE *startp = p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4860 const Py_UNICODE *endp = p + size;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4861 /* pointer into the output */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4862 Py_UNICODE *str;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4863 /* current output position */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4864 Py_ssize_t respos = 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4865 char *reason = "character maps to <undefined>";
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4866 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4867 PyObject *exc = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4868 /* the following variable is used for caching string comparisons
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4869 * -1=not initialized, 0=unknown, 1=strict, 2=replace,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4870 * 3=ignore, 4=xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4871 int known_errorHandler = -1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4872
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4873 if (mapping == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4874 PyErr_BadArgument();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4875 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4876 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4877
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4878 /* allocate enough for a simple 1:1 translation without
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4879 replacements, if we need more, we'll resize */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4880 res = PyUnicode_FromUnicode(NULL, size);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4881 if (res == NULL)
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4882 goto onError;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4883 if (size == 0)
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4884 return res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4885 str = PyUnicode_AS_UNICODE(res);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4886
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4887 while (p<endp) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4888 /* try to encode it */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4889 PyObject *x = NULL;
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4890 if (charmaptranslate_output(startp, p, size, mapping, &res, &str, &x)) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4891 Py_XDECREF(x);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4892 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4893 }
27774
bfc524379307 Fix two refcounting bugs
Walter Dörwald <walter@livinglogic.de>
parents: 27555
diff changeset
4894 Py_XDECREF(x);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4895 if (x!=Py_None) /* it worked => adjust input pointer */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4896 ++p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4897 else { /* untranslatable character */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4898 PyObject *repunicode = NULL; /* initialize to prevent gcc warning */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4899 Py_ssize_t repsize;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
4900 Py_ssize_t newpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4901 Py_UNICODE *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4902 /* startpos for collecting untranslatable chars */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4903 const Py_UNICODE *collstart = p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4904 const Py_UNICODE *collend = p+1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4905 const Py_UNICODE *coll;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4906
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4907 /* find all untranslatable characters */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4908 while (collend < endp) {
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4909 if (charmaptranslate_lookup(*collend, mapping, &x))
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4910 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4911 Py_XDECREF(x);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4912 if (x!=Py_None)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4913 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4914 ++collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4915 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4916 /* cache callback name lookup
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4917 * (if not done yet, i.e. it's the first error) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4918 if (known_errorHandler==-1) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4919 if ((errors==NULL) || (!strcmp(errors, "strict")))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4920 known_errorHandler = 1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4921 else if (!strcmp(errors, "replace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4922 known_errorHandler = 2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4923 else if (!strcmp(errors, "ignore"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4924 known_errorHandler = 3;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4925 else if (!strcmp(errors, "xmlcharrefreplace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4926 known_errorHandler = 4;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4927 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4928 known_errorHandler = 0;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4929 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4930 switch (known_errorHandler) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4931 case 1: /* strict */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4932 raise_translate_exception(&exc, startp, size, collstart-startp, collend-startp, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4933 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4934 case 2: /* replace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4935 /* No need to check for space, this is a 1:1 replacement */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4936 for (coll = collstart; coll<collend; ++coll)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4937 *str++ = '?';
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4938 /* fall through */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4939 case 3: /* ignore */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4940 p = collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4941 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4942 case 4: /* xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4943 /* generate replacement (temporarily (mis)uses p) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4944 for (p = collstart; p < collend; ++p) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4945 char buffer[2+29+1+1];
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4946 char *cp;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4947 sprintf(buffer, "&#%d;", (int)*p);
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4948 if (charmaptranslate_makespace(&res, &str,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4949 (str-PyUnicode_AS_UNICODE(res))+strlen(buffer)+(endp-collend)))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4950 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4951 for (cp = buffer; *cp; ++cp)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4952 *str++ = *cp;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4953 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4954 p = collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4955 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4956 default:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4957 repunicode = unicode_translate_call_errorhandler(errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4958 reason, startp, size, &exc,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4959 collstart-startp, collend-startp, &newpos);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4960 if (repunicode == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4961 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4962 /* generate replacement */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4963 repsize = PyUnicode_GET_SIZE(repunicode);
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4964 if (charmaptranslate_makespace(&res, &str,
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4965 (str-PyUnicode_AS_UNICODE(res))+repsize+(endp-collend))) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4966 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4967 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4968 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4969 for (uni2 = PyUnicode_AS_UNICODE(repunicode); repsize-->0; ++uni2)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4970 *str++ = *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4971 p = startp + newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4972 Py_DECREF(repunicode);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4973 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4974 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4975 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4976 /* Resize if we allocated to much */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4977 respos = str-PyUnicode_AS_UNICODE(res);
30765
fe3f5592305d Fix a bug in the memory reallocation code of PyUnicode_TranslateCharmap().
Walter Dörwald <walter@livinglogic.de>
parents: 30707
diff changeset
4978 if (respos<PyUnicode_GET_SIZE(res)) {
30431
cd0c3f916b34 Change checks of PyUnicode_Resize() return value for clarity.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 30216
diff changeset
4979 if (_PyUnicode_Resize(&res, respos) < 0)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4980 goto onError;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4981 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4982 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4983 Py_XDECREF(errorHandler);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4984 return res;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4985
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4986 onError:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4987 Py_XDECREF(res);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4988 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
4989 Py_XDECREF(errorHandler);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4990 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4991 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4992
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4993 PyObject *PyUnicode_Translate(PyObject *str,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4994 PyObject *mapping,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4995 const char *errors)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4996 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4997 PyObject *result;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
4998
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4999 str = PyUnicode_FromObject(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5000 if (str == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5001 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5002 result = PyUnicode_TranslateCharmap(PyUnicode_AS_UNICODE(str),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5003 PyUnicode_GET_SIZE(str),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5004 mapping,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5005 errors);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5006 Py_DECREF(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5007 return result;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5008
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5009 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5010 Py_XDECREF(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5011 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5012 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5013
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5014 /* --- Decimal Encoder ---------------------------------------------------- */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5015
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5016 int PyUnicode_EncodeDecimal(Py_UNICODE *s,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5017 Py_ssize_t length,
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5018 char *output,
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5019 const char *errors)
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5020 {
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5021 Py_UNICODE *p, *end;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5022 PyObject *errorHandler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5023 PyObject *exc = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5024 const char *encoding = "decimal";
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5025 const char *reason = "invalid decimal Unicode string";
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5026 /* the following variable is used for caching string comparisons
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5027 * -1=not initialized, 0=unknown, 1=strict, 2=replace, 3=ignore, 4=xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5028 int known_errorHandler = -1;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5029
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5030 if (output == NULL) {
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5031 PyErr_BadArgument();
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5032 return -1;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5033 }
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5034
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5035 p = s;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5036 end = s + length;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5037 while (p < end) {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5038 register Py_UNICODE ch = *p;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5039 int decimal;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5040 PyObject *repunicode;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5041 Py_ssize_t repsize;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5042 Py_ssize_t newpos;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5043 Py_UNICODE *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5044 Py_UNICODE *collstart;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5045 Py_UNICODE *collend;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5046
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5047 if (Py_UNICODE_ISSPACE(ch)) {
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5048 *output++ = ' ';
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5049 ++p;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5050 continue;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5051 }
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5052 decimal = Py_UNICODE_TODECIMAL(ch);
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5053 if (decimal >= 0) {
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5054 *output++ = '0' + decimal;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5055 ++p;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5056 continue;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5057 }
12275
cbfa95a6a03e Conrad Huang points out that "if (0 < ch < 256)", while legal C,
Guido van Rossum <guido@python.org>
parents: 12249
diff changeset
5058 if (0 < ch && ch < 256) {
12577
85f00f8cd674 Fix warning detected by VC++ on assignment of Py_UNICODE to char.
Guido van Rossum <guido@python.org>
parents: 12576
diff changeset
5059 *output++ = (char)ch;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5060 ++p;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5061 continue;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5062 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5063 /* All other characters are considered unencodable */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5064 collstart = p;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5065 collend = p+1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5066 while (collend < end) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5067 if ((0 < *collend && *collend < 256) ||
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5068 !Py_UNICODE_ISSPACE(*collend) ||
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5069 Py_UNICODE_TODECIMAL(*collend))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5070 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5071 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5072 /* cache callback name lookup
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5073 * (if not done yet, i.e. it's the first error) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5074 if (known_errorHandler==-1) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5075 if ((errors==NULL) || (!strcmp(errors, "strict")))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5076 known_errorHandler = 1;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5077 else if (!strcmp(errors, "replace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5078 known_errorHandler = 2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5079 else if (!strcmp(errors, "ignore"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5080 known_errorHandler = 3;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5081 else if (!strcmp(errors, "xmlcharrefreplace"))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5082 known_errorHandler = 4;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5083 else
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5084 known_errorHandler = 0;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5085 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5086 switch (known_errorHandler) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5087 case 1: /* strict */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5088 raise_encode_exception(&exc, encoding, s, length, collstart-s, collend-s, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5089 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5090 case 2: /* replace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5091 for (p = collstart; p < collend; ++p)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5092 *output++ = '?';
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5093 /* fall through */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5094 case 3: /* ignore */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5095 p = collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5096 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5097 case 4: /* xmlcharrefreplace */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5098 /* generate replacement (temporarily (mis)uses p) */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5099 for (p = collstart; p < collend; ++p)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5100 output += sprintf(output, "&#%d;", (int)*p);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5101 p = collend;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5102 break;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5103 default:
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5104 repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5105 encoding, reason, s, length, &exc,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5106 collstart-s, collend-s, &newpos);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5107 if (repunicode == NULL)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5108 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5109 /* generate replacement */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5110 repsize = PyUnicode_GET_SIZE(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5111 for (uni2 = PyUnicode_AS_UNICODE(repunicode); repsize-->0; ++uni2) {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5112 Py_UNICODE ch = *uni2;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5113 if (Py_UNICODE_ISSPACE(ch))
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5114 *output++ = ' ';
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5115 else {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5116 decimal = Py_UNICODE_TODECIMAL(ch);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5117 if (decimal >= 0)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5118 *output++ = '0' + decimal;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5119 else if (0 < ch && ch < 256)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5120 *output++ = (char)ch;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5121 else {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5122 Py_DECREF(repunicode);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5123 raise_encode_exception(&exc, encoding,
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5124 s, length, collstart-s, collend-s, reason);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5125 goto onError;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5126 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5127 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5128 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5129 p = s + newpos;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5130 Py_DECREF(repunicode);
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5131 }
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5132 }
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5133 /* 0-terminate the output string */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5134 *output++ = '\0';
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5135 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5136 Py_XDECREF(errorHandler);
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5137 return 0;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5138
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5139 onError:
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5140 Py_XDECREF(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
5141 Py_XDECREF(errorHandler);
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5142 return -1;
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5143 }
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12193
diff changeset
5144
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5145 /* --- Helpers ------------------------------------------------------------ */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5146
45381
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
5147 #include "stringlib/unicodedefs.h"
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
5148
43888
1fb353776dfb Made _ParseTupleFinds only defined to unicodeobject.c
Facundo Batista <facundobatista@gmail.com>
parents: 43886
diff changeset
5149 #define FROM_UNICODE
38383
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
5150
38381
b32f9bb2e077 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38347
diff changeset
5151 #include "stringlib/fastsearch.h"
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5152
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5153 #include "stringlib/count.h"
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5154 #include "stringlib/find.h"
38383
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
5155 #include "stringlib/partition.h"
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
5156
38402
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5157 /* helper macro to fixup start/end slice values */
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5158 #define FIX_START_END(obj) \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5159 if (start < 0) \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5160 start += (obj)->length; \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5161 if (start < 0) \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5162 start = 0; \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5163 if (end > (obj)->length) \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5164 end = (obj)->length; \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5165 if (end < 0) \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5166 end += (obj)->length; \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5167 if (end < 0) \
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5168 end = 0;
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5169
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5170 Py_ssize_t PyUnicode_Count(PyObject *str,
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5171 PyObject *substr,
38347
6d8615f9f1cc use Py_LOCAL also for string and unicode objects
Fredrik Lundh <fredrik@pythonware.com>
parents: 38339
diff changeset
5172 Py_ssize_t start,
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5173 Py_ssize_t end)
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5174 {
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5175 Py_ssize_t result;
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5176 PyUnicodeObject* str_obj;
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5177 PyUnicodeObject* sub_obj;
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5178
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5179 str_obj = (PyUnicodeObject*) PyUnicode_FromObject(str);
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5180 if (!str_obj)
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5181 return -1;
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5182 sub_obj = (PyUnicodeObject*) PyUnicode_FromObject(substr);
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5183 if (!sub_obj) {
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5184 Py_DECREF(str_obj);
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5185 return -1;
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5186 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5187
38402
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5188 FIX_START_END(str_obj);
15910
f7f3be3d00b9 Added checks to prevent PyUnicode_Count() from dumping core
Marc-André Lemburg <mal@egenix.com>
parents: 15749
diff changeset
5189
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5190 result = stringlib_count(
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5191 str_obj->str + start, end - start, sub_obj->str, sub_obj->length
38275
43b1474b512b needforspeed: use "fastsearch" for count and findstring helpers. this
Fredrik Lundh <fredrik@pythonware.com>
parents: 38274
diff changeset
5192 );
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
5193
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5194 Py_DECREF(sub_obj);
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5195 Py_DECREF(str_obj);
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
5196
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5197 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5198 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5199
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5200 Py_ssize_t PyUnicode_Find(PyObject *str,
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5201 PyObject *sub,
38401
adc214fc1e1c needforspeed: stringlib refactoring: use stringlib/find for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 38400
diff changeset
5202 Py_ssize_t start,
adc214fc1e1c needforspeed: stringlib refactoring: use stringlib/find for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 38400
diff changeset
5203 Py_ssize_t end,
adc214fc1e1c needforspeed: stringlib refactoring: use stringlib/find for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 38400
diff changeset
5204 int direction)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5205 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5206 Py_ssize_t result;
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5207
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5208 str = PyUnicode_FromObject(str);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5209 if (!str)
23561
c5e8f89fd358 Fix for bug [ 561796 ] string.find causes lazy error
Marc-André Lemburg <mal@egenix.com>
parents: 23542
diff changeset
5210 return -2;
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5211 sub = PyUnicode_FromObject(sub);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5212 if (!sub) {
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5213 Py_DECREF(str);
23561
c5e8f89fd358 Fix for bug [ 561796 ] string.find causes lazy error
Marc-André Lemburg <mal@egenix.com>
parents: 23542
diff changeset
5214 return -2;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5215 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5216
38401
adc214fc1e1c needforspeed: stringlib refactoring: use stringlib/find for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 38400
diff changeset
5217 if (direction > 0)
38441
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5218 result = stringlib_find_slice(
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5219 PyUnicode_AS_UNICODE(str), PyUnicode_GET_SIZE(str),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5220 PyUnicode_AS_UNICODE(sub), PyUnicode_GET_SIZE(sub),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5221 start, end
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5222 );
38401
adc214fc1e1c needforspeed: stringlib refactoring: use stringlib/find for unicode
Fredrik Lundh <fredrik@pythonware.com>
parents: 38400
diff changeset
5223 else
38441
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5224 result = stringlib_rfind_slice(
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5225 PyUnicode_AS_UNICODE(str), PyUnicode_GET_SIZE(str),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5226 PyUnicode_AS_UNICODE(sub), PyUnicode_GET_SIZE(sub),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5227 start, end
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
5228 );
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5229
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5230 Py_DECREF(str);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5231 Py_DECREF(sub);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
5232
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5233 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5234 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5235
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5236 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5237 int tailmatch(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5238 PyUnicodeObject *substring,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5239 Py_ssize_t start,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5240 Py_ssize_t end,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5241 int direction)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5242 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5243 if (substring->length == 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5244 return 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5245
38402
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
5246 FIX_START_END(self);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5247
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5248 end -= substring->length;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5249 if (end < start)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5250 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5251
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5252 if (direction > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5253 if (Py_UNICODE_MATCH(self, end, substring))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5254 return 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5255 } else {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5256 if (Py_UNICODE_MATCH(self, start, substring))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5257 return 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5258 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5259
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5260 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5261 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5262
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5263 Py_ssize_t PyUnicode_Tailmatch(PyObject *str,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5264 PyObject *substr,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5265 Py_ssize_t start,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5266 Py_ssize_t end,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5267 int direction)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5268 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5269 Py_ssize_t result;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5270
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5271 str = PyUnicode_FromObject(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5272 if (str == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5273 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5274 substr = PyUnicode_FromObject(substr);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5275 if (substr == NULL) {
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
5276 Py_DECREF(str);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5277 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5278 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5279
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5280 result = tailmatch((PyUnicodeObject *)str,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5281 (PyUnicodeObject *)substr,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5282 start, end, direction);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5283 Py_DECREF(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5284 Py_DECREF(substr);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5285 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5286 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5287
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5288 /* Apply fixfct filter to the Unicode object self and return a
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5289 reference to the modified object */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5290
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5291 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5292 PyObject *fixup(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5293 int (*fixfct)(PyUnicodeObject *s))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5294 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5295
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5296 PyUnicodeObject *u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5297
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
5298 u = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, self->length);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5299 if (u == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5300 return NULL;
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
5301
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
5302 Py_UNICODE_COPY(u->str, self->str, self->length);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
5303
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
5304 if (!fixfct(u) && PyUnicode_CheckExact(self)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5305 /* fixfct should return TRUE if it modified the buffer. If
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5306 FALSE, return a reference to the original buffer instead
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5307 (to save space, not time) */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5308 Py_INCREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5309 Py_DECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5310 return (PyObject*) self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5311 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5312 return (PyObject*) u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5313 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5314
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5315 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5316 int fixupper(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5317 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5318 Py_ssize_t len = self->length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5319 Py_UNICODE *s = self->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5320 int status = 0;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5321
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5322 while (len-- > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5323 register Py_UNICODE ch;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5324
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5325 ch = Py_UNICODE_TOUPPER(*s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5326 if (ch != *s) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5327 status = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5328 *s = ch;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5329 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5330 s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5331 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5332
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5333 return status;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5334 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5335
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5336 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5337 int fixlower(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5338 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5339 Py_ssize_t len = self->length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5340 Py_UNICODE *s = self->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5341 int status = 0;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5342
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5343 while (len-- > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5344 register Py_UNICODE ch;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5345
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5346 ch = Py_UNICODE_TOLOWER(*s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5347 if (ch != *s) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5348 status = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5349 *s = ch;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5350 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5351 s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5352 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5353
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5354 return status;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5355 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5356
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5357 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5358 int fixswapcase(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5359 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5360 Py_ssize_t len = self->length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5361 Py_UNICODE *s = self->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5362 int status = 0;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5363
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5364 while (len-- > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5365 if (Py_UNICODE_ISUPPER(*s)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5366 *s = Py_UNICODE_TOLOWER(*s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5367 status = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5368 } else if (Py_UNICODE_ISLOWER(*s)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5369 *s = Py_UNICODE_TOUPPER(*s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5370 status = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5371 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5372 s++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5373 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5374
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5375 return status;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5376 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5377
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5378 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5379 int fixcapitalize(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5380 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5381 Py_ssize_t len = self->length;
16332
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5382 Py_UNICODE *s = self->str;
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5383 int status = 0;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5384
16332
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5385 if (len == 0)
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5386 return 0;
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5387 if (Py_UNICODE_ISLOWER(*s)) {
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5388 *s = Py_UNICODE_TOUPPER(*s);
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5389 status = 1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5390 }
16332
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5391 s++;
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5392 while (--len > 0) {
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5393 if (Py_UNICODE_ISUPPER(*s)) {
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5394 *s = Py_UNICODE_TOLOWER(*s);
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5395 status = 1;
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5396 }
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5397 s++;
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5398 }
bacbb45fe844 Fixed .capitalize() method of Unicode objects to work like the
Marc-André Lemburg <mal@egenix.com>
parents: 16226
diff changeset
5399 return status;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5400 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5401
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5402 static
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5403 int fixtitle(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5404 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5405 register Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5406 register Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5407 int previous_is_cased;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5408
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5409 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5410 if (PyUnicode_GET_SIZE(self) == 1) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5411 Py_UNICODE ch = Py_UNICODE_TOTITLE(*p);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5412 if (*p != ch) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5413 *p = ch;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5414 return 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5415 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5416 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5417 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5418 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5419
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5420 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5421 previous_is_cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5422 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5423 register const Py_UNICODE ch = *p;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5424
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5425 if (previous_is_cased)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5426 *p = Py_UNICODE_TOLOWER(ch);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5427 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5428 *p = Py_UNICODE_TOTITLE(ch);
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5429
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5430 if (Py_UNICODE_ISLOWER(ch) ||
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5431 Py_UNICODE_ISUPPER(ch) ||
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5432 Py_UNICODE_ISTITLE(ch))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5433 previous_is_cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5434 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5435 previous_is_cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5436 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5437 return 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5438 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5439
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5440 PyObject *
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5441 PyUnicode_Join(PyObject *separator, PyObject *seq)
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5442 {
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5443 PyObject *internal_separator = NULL;
33499
a4d3673bcd66 Initialize sep and seplen to suppress warning from gcc.
Skip Montanaro <skip@pobox.com>
parents: 33495
diff changeset
5444 const Py_UNICODE blank = ' ';
a4d3673bcd66 Initialize sep and seplen to suppress warning from gcc.
Skip Montanaro <skip@pobox.com>
parents: 33495
diff changeset
5445 const Py_UNICODE *sep = &blank;
38240
0373a074004c PyUnicode_Join(): Recent code changes introduced new
Tim Peters <tim.peters@gmail.com>
parents: 38239
diff changeset
5446 Py_ssize_t seplen = 1;
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5447 PyUnicodeObject *res = NULL; /* the result */
38240
0373a074004c PyUnicode_Join(): Recent code changes introduced new
Tim Peters <tim.peters@gmail.com>
parents: 38239
diff changeset
5448 Py_ssize_t res_alloc = 100; /* # allocated bytes for string in res */
0373a074004c PyUnicode_Join(): Recent code changes introduced new
Tim Peters <tim.peters@gmail.com>
parents: 38239
diff changeset
5449 Py_ssize_t res_used; /* # used bytes */
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5450 Py_UNICODE *res_p; /* pointer to free byte in res's string area */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5451 PyObject *fseq; /* PySequence_Fast(seq) */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5452 Py_ssize_t seqlen; /* len(fseq) -- number of items in sequence */
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5453 PyObject *item;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5454 Py_ssize_t i;
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5455
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5456 fseq = PySequence_Fast(seq, "");
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5457 if (fseq == NULL) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5458 return NULL;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5459 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5460
33311
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5461 /* Grrrr. A codec may be invoked to convert str objects to
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5462 * Unicode, and so it's possible to call back into Python code
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5463 * during PyUnicode_FromObject(), and so it's possible for a sick
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5464 * codec to change the size of fseq (if seq is a list). Therefore
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5465 * we have to keep refetching the size -- can't assume seqlen
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5466 * is invariant.
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5467 */
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5468 seqlen = PySequence_Fast_GET_SIZE(fseq);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5469 /* If empty sequence, return u"". */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5470 if (seqlen == 0) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5471 res = _PyUnicode_New(0); /* empty sequence; return u"" */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5472 goto Done;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5473 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5474 /* If singleton sequence with an exact Unicode, return that. */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5475 if (seqlen == 1) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5476 item = PySequence_Fast_GET_ITEM(fseq, 0);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5477 if (PyUnicode_CheckExact(item)) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5478 Py_INCREF(item);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5479 res = (PyUnicodeObject *)item;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5480 goto Done;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5481 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5482 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5483
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5484 /* At least two items to join, or one that isn't exact Unicode. */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5485 if (seqlen > 1) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5486 /* Set up sep and seplen -- they're needed. */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5487 if (separator == NULL) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5488 sep = &blank;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5489 seplen = 1;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5490 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5491 else {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5492 internal_separator = PyUnicode_FromObject(separator);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5493 if (internal_separator == NULL)
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5494 goto onError;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5495 sep = PyUnicode_AS_UNICODE(internal_separator);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5496 seplen = PyUnicode_GET_SIZE(internal_separator);
33311
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5497 /* In case PyUnicode_FromObject() mutated seq. */
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5498 seqlen = PySequence_Fast_GET_SIZE(fseq);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5499 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5500 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5501
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5502 /* Get space. */
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5503 res = _PyUnicode_New(res_alloc);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5504 if (res == NULL)
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5505 goto onError;
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5506 res_p = PyUnicode_AS_UNICODE(res);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5507 res_used = 0;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5508
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5509 for (i = 0; i < seqlen; ++i) {
38240
0373a074004c PyUnicode_Join(): Recent code changes introduced new
Tim Peters <tim.peters@gmail.com>
parents: 38239
diff changeset
5510 Py_ssize_t itemlen;
0373a074004c PyUnicode_Join(): Recent code changes introduced new
Tim Peters <tim.peters@gmail.com>
parents: 38239
diff changeset
5511 Py_ssize_t new_res_used;
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5512
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5513 item = PySequence_Fast_GET_ITEM(fseq, i);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5514 /* Convert item to Unicode. */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
5515 if (! PyUnicode_Check(item) && ! PyString_Check(item)) {
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5516 PyErr_Format(PyExc_TypeError,
37775
d8b23ba4a274 Use %zd instead of %i as format character (in call to PyErr_Format) for
Thomas Wouters <thomas@python.org>
parents: 37702
diff changeset
5517 "sequence item %zd: expected string or Unicode,"
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5518 " %.80s found",
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
5519 i, Py_TYPE(item)->tp_name);
17687
9e093bf8f6a9 Make unicode.join() work nice with iterators. This also required a change
Tim Peters <tim.peters@gmail.com>
parents: 17640
diff changeset
5520 goto onError;
9e093bf8f6a9 Make unicode.join() work nice with iterators. This also required a change
Tim Peters <tim.peters@gmail.com>
parents: 17640
diff changeset
5521 }
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5522 item = PyUnicode_FromObject(item);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5523 if (item == NULL)
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5524 goto onError;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5525 /* We own a reference to item from here on. */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5526
33311
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5527 /* In case PyUnicode_FromObject() mutated seq. */
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5528 seqlen = PySequence_Fast_GET_SIZE(fseq);
59bd121eeeb7 PyUnicode_Join(): Bozo Alert. While this is chugging along, it may
Tim Peters <tim.peters@gmail.com>
parents: 33310
diff changeset
5529
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5530 /* Make sure we have enough space for the separator and the item. */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5531 itemlen = PyUnicode_GET_SIZE(item);
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5532 new_res_used = res_used + itemlen;
38709
67450db26f81 Apply perky's fix for #1503157: "/".join([u"", u""]) raising OverflowError.
Georg Brandl <georg@python.org>
parents: 38697
diff changeset
5533 if (new_res_used < 0)
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5534 goto Overflow;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5535 if (i < seqlen - 1) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5536 new_res_used += seplen;
38709
67450db26f81 Apply perky's fix for #1503157: "/".join([u"", u""]) raising OverflowError.
Georg Brandl <georg@python.org>
parents: 38697
diff changeset
5537 if (new_res_used < 0)
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5538 goto Overflow;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5539 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5540 if (new_res_used > res_alloc) {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5541 /* double allocated size until it's big enough */
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5542 do {
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5543 res_alloc += res_alloc;
38240
0373a074004c PyUnicode_Join(): Recent code changes introduced new
Tim Peters <tim.peters@gmail.com>
parents: 38239
diff changeset
5544 if (res_alloc <= 0)
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5545 goto Overflow;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5546 } while (new_res_used > res_alloc);
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5547 if (_PyUnicode_Resize(&res, res_alloc) < 0) {
20007
56ffee1eeeec Fix Unicode .join() method to raise a TypeError for sequence
Marc-André Lemburg <mal@egenix.com>
parents: 19999
diff changeset
5548 Py_DECREF(item);
56ffee1eeeec Fix Unicode .join() method to raise a TypeError for sequence
Marc-André Lemburg <mal@egenix.com>
parents: 19999
diff changeset
5549 goto onError;
56ffee1eeeec Fix Unicode .join() method to raise a TypeError for sequence
Marc-André Lemburg <mal@egenix.com>
parents: 19999
diff changeset
5550 }
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5551 res_p = PyUnicode_AS_UNICODE(res) + res_used;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5552 }
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5553
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5554 /* Copy item, and maybe the separator. */
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5555 Py_UNICODE_COPY(res_p, PyUnicode_AS_UNICODE(item), itemlen);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5556 res_p += itemlen;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5557 if (i < seqlen - 1) {
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5558 Py_UNICODE_COPY(res_p, sep, seplen);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5559 res_p += seplen;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5560 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5561 Py_DECREF(item);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5562 res_used = new_res_used;
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5563 }
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5564
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5565 /* Shrink res to match the used area; this probably can't fail,
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5566 * but it's cheap to check.
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5567 */
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5568 if (_PyUnicode_Resize(&res, res_used) < 0)
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5569 goto onError;
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5570
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5571 Done:
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5572 Py_XDECREF(internal_separator);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5573 Py_DECREF(fseq);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5574 return (PyObject *)res;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5575
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5576 Overflow:
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5577 PyErr_SetString(PyExc_OverflowError,
38709
67450db26f81 Apply perky's fix for #1503157: "/".join([u"", u""]) raising OverflowError.
Georg Brandl <georg@python.org>
parents: 38697
diff changeset
5578 "join() result is too long for a Python string");
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5579 Py_DECREF(item);
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5580 /* fall through */
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5581
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5582 onError:
33300
77de38534d26 PyUnicode_Join(): Two primary aims:
Tim Peters <tim.peters@gmail.com>
parents: 32951
diff changeset
5583 Py_XDECREF(internal_separator);
33310
8579859f198c PyUnicode_Join(): Rewrote to use PySequence_Fast(). This doesn't do
Tim Peters <tim.peters@gmail.com>
parents: 33304
diff changeset
5584 Py_DECREF(fseq);
17687
9e093bf8f6a9 Make unicode.join() work nice with iterators. This also required a change
Tim Peters <tim.peters@gmail.com>
parents: 17640
diff changeset
5585 Py_XDECREF(res);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5586 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5587 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5588
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5589 static
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5590 PyUnicodeObject *pad(PyUnicodeObject *self,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5591 Py_ssize_t left,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5592 Py_ssize_t right,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5593 Py_UNICODE fill)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5594 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5595 PyUnicodeObject *u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5596
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5597 if (left < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5598 left = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5599 if (right < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5600 right = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5601
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
5602 if (left == 0 && right == 0 && PyUnicode_CheckExact(self)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5603 Py_INCREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5604 return self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5605 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5606
48114
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
5607 if (left > PY_SSIZE_T_MAX - self->length ||
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
5608 right > PY_SSIZE_T_MAX - (left + self->length)) {
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
5609 PyErr_SetString(PyExc_OverflowError, "padded string is too long");
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
5610 return NULL;
c4d24fc8ba8a Security patches from Apple: prevent int overflow when allocating memory
Neal Norwitz <nnorwitz@gmail.com>
parents: 48070
diff changeset
5611 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5612 u = _PyUnicode_New(left + self->length + right);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5613 if (u) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5614 if (left)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5615 Py_UNICODE_FILL(u->str, fill, left);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5616 Py_UNICODE_COPY(u->str + left, self->str, self->length);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5617 if (right)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5618 Py_UNICODE_FILL(u->str + left + self->length, fill, right);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5619 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5620
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5621 return u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5622 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5623
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5624 #define SPLIT_APPEND(data, left, right) \
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5625 str = PyUnicode_FromUnicode((data) + (left), (right) - (left)); \
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5626 if (!str) \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5627 goto onError; \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5628 if (PyList_Append(list, str)) { \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5629 Py_DECREF(str); \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5630 goto onError; \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5631 } \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5632 else \
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5633 Py_DECREF(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5634
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5635 static
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5636 PyObject *split_whitespace(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5637 PyObject *list,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5638 Py_ssize_t maxcount)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5639 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5640 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5641 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5642 Py_ssize_t len = self->length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5643 PyObject *str;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5644 register const Py_UNICODE *buf = self->str;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5645
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5646 for (i = j = 0; i < len; ) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5647 /* find a token */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5648 while (i < len && Py_UNICODE_ISSPACE(buf[i]))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5649 i++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5650 j = i;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5651 while (i < len && !Py_UNICODE_ISSPACE(buf[i]))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5652 i++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5653 if (j < i) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5654 if (maxcount-- <= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5655 break;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5656 SPLIT_APPEND(buf, j, i);
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5657 while (i < len && Py_UNICODE_ISSPACE(buf[i]))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5658 i++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5659 j = i;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5660 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5661 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5662 if (j < len) {
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5663 SPLIT_APPEND(buf, j, len);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5664 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5665 return list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5666
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5667 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5668 Py_DECREF(list);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5669 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5670 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5671
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5672 PyObject *PyUnicode_Splitlines(PyObject *string,
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
5673 int keepends)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5674 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5675 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5676 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5677 Py_ssize_t len;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5678 PyObject *list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5679 PyObject *str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5680 Py_UNICODE *data;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5681
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5682 string = PyUnicode_FromObject(string);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5683 if (string == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5684 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5685 data = PyUnicode_AS_UNICODE(string);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5686 len = PyUnicode_GET_SIZE(string);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5687
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5688 list = PyList_New(0);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5689 if (!list)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5690 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5691
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5692 for (i = j = 0; i < len; ) {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5693 Py_ssize_t eol;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5694
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5695 /* Find a line and append it */
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5696 while (i < len && !BLOOM_LINEBREAK(data[i]))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5697 i++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5698
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5699 /* Skip the line break reading CRLF as one line break */
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
5700 eol = i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5701 if (i < len) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5702 if (data[i] == '\r' && i + 1 < len &&
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5703 data[i+1] == '\n')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5704 i += 2;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5705 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5706 i++;
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
5707 if (keepends)
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
5708 eol = i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5709 }
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
5710 SPLIT_APPEND(data, j, eol);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5711 j = i;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5712 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5713 if (j < len) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5714 SPLIT_APPEND(data, j, len);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5715 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5716
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5717 Py_DECREF(string);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5718 return list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5719
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5720 onError:
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
5721 Py_XDECREF(list);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5722 Py_DECREF(string);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5723 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5724 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5725
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5726 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5727 PyObject *split_char(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5728 PyObject *list,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5729 Py_UNICODE ch,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5730 Py_ssize_t maxcount)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5731 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5732 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5733 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5734 Py_ssize_t len = self->length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5735 PyObject *str;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5736 register const Py_UNICODE *buf = self->str;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5737
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5738 for (i = j = 0; i < len; ) {
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5739 if (buf[i] == ch) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5740 if (maxcount-- <= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5741 break;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5742 SPLIT_APPEND(buf, j, i);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5743 i = j = i + 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5744 } else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5745 i++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5746 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5747 if (j <= len) {
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5748 SPLIT_APPEND(buf, j, len);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5749 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5750 return list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5751
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5752 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5753 Py_DECREF(list);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5754 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5755 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5756
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5757 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5758 PyObject *split_substring(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5759 PyObject *list,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5760 PyUnicodeObject *substring,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5761 Py_ssize_t maxcount)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5762 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5763 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5764 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5765 Py_ssize_t len = self->length;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5766 Py_ssize_t sublen = substring->length;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5767 PyObject *str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5768
15609
0906c8e2c961 Fix off-by-one error in split_substring(). Fixes SF bug #122162.
Guido van Rossum <guido@python.org>
parents: 15579
diff changeset
5769 for (i = j = 0; i <= len - sublen; ) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5770 if (Py_UNICODE_MATCH(self, i, substring)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5771 if (maxcount-- <= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5772 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5773 SPLIT_APPEND(self->str, j, i);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5774 i = j = i + sublen;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5775 } else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5776 i++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5777 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5778 if (j <= len) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5779 SPLIT_APPEND(self->str, j, len);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5780 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5781 return list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5782
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5783 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5784 Py_DECREF(list);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5785 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5786 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5787
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5788 static
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5789 PyObject *rsplit_whitespace(PyUnicodeObject *self,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5790 PyObject *list,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5791 Py_ssize_t maxcount)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5792 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5793 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5794 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5795 Py_ssize_t len = self->length;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5796 PyObject *str;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5797 register const Py_UNICODE *buf = self->str;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5798
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5799 for (i = j = len - 1; i >= 0; ) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5800 /* find a token */
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5801 while (i >= 0 && Py_UNICODE_ISSPACE(buf[i]))
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5802 i--;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5803 j = i;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5804 while (i >= 0 && !Py_UNICODE_ISSPACE(buf[i]))
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5805 i--;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5806 if (j > i) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5807 if (maxcount-- <= 0)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5808 break;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5809 SPLIT_APPEND(buf, i + 1, j + 1);
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5810 while (i >= 0 && Py_UNICODE_ISSPACE(buf[i]))
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5811 i--;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5812 j = i;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5813 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5814 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5815 if (j >= 0) {
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5816 SPLIT_APPEND(buf, 0, j + 1);
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5817 }
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5818 if (PyList_Reverse(list) < 0)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5819 goto onError;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5820 return list;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5821
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5822 onError:
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5823 Py_DECREF(list);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5824 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5825 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5826
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5827 static
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5828 PyObject *rsplit_char(PyUnicodeObject *self,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5829 PyObject *list,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5830 Py_UNICODE ch,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5831 Py_ssize_t maxcount)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5832 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5833 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5834 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5835 Py_ssize_t len = self->length;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5836 PyObject *str;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5837 register const Py_UNICODE *buf = self->str;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5838
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5839 for (i = j = len - 1; i >= 0; ) {
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5840 if (buf[i] == ch) {
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5841 if (maxcount-- <= 0)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5842 break;
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5843 SPLIT_APPEND(buf, i + 1, j + 1);
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5844 j = i = i - 1;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5845 } else
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5846 i--;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5847 }
31252
892116a886e0 Fix unicode.rsplit()'s bug that ignores separater on the end of string when
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31248
diff changeset
5848 if (j >= -1) {
45053
26278796d4da Patch #1970 by Antoine Pitrou: Speedup unicode whitespace and linebreak detection. The speedup is about 25% for split() (571 / 457 usec) and 35% (175 / 127 usec )for splitlines()
Christian Heimes <christian@cheimes.de>
parents: 44945
diff changeset
5849 SPLIT_APPEND(buf, 0, j + 1);
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5850 }
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5851 if (PyList_Reverse(list) < 0)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5852 goto onError;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5853 return list;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5854
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5855 onError:
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5856 Py_DECREF(list);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5857 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5858 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5859
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5860 static
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5861 PyObject *rsplit_substring(PyUnicodeObject *self,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5862 PyObject *list,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5863 PyUnicodeObject *substring,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5864 Py_ssize_t maxcount)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5865 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5866 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5867 register Py_ssize_t j;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5868 Py_ssize_t len = self->length;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5869 Py_ssize_t sublen = substring->length;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5870 PyObject *str;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5871
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5872 for (i = len - sublen, j = len; i >= 0; ) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5873 if (Py_UNICODE_MATCH(self, i, substring)) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5874 if (maxcount-- <= 0)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5875 break;
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5876 SPLIT_APPEND(self->str, i + sublen, j);
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5877 j = i;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5878 i -= sublen;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5879 } else
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5880 i--;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5881 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5882 if (j >= 0) {
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5883 SPLIT_APPEND(self->str, 0, j);
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5884 }
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5885 if (PyList_Reverse(list) < 0)
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
5886 goto onError;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5887 return list;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5888
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5889 onError:
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5890 Py_DECREF(list);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5891 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5892 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5893
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5894 #undef SPLIT_APPEND
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5895
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5896 static
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5897 PyObject *split(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5898 PyUnicodeObject *substring,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5899 Py_ssize_t maxcount)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5900 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5901 PyObject *list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5902
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5903 if (maxcount < 0)
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5904 maxcount = PY_SSIZE_T_MAX;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5905
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5906 list = PyList_New(0);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5907 if (!list)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5908 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5909
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5910 if (substring == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5911 return split_whitespace(self,list,maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5912
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5913 else if (substring->length == 1)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5914 return split_char(self,list,substring->str[0],maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5915
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5916 else if (substring->length == 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5917 Py_DECREF(list);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5918 PyErr_SetString(PyExc_ValueError, "empty separator");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5919 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5920 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5921 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5922 return split_substring(self,list,substring,maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5923 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5924
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
5925 static
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5926 PyObject *rsplit(PyUnicodeObject *self,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5927 PyUnicodeObject *substring,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5928 Py_ssize_t maxcount)
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5929 {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5930 PyObject *list;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5931
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5932 if (maxcount < 0)
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5933 maxcount = PY_SSIZE_T_MAX;
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5934
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5935 list = PyList_New(0);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5936 if (!list)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5937 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5938
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5939 if (substring == NULL)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5940 return rsplit_whitespace(self,list,maxcount);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5941
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5942 else if (substring->length == 1)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5943 return rsplit_char(self,list,substring->str[0],maxcount);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5944
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5945 else if (substring->length == 0) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5946 Py_DECREF(list);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5947 PyErr_SetString(PyExc_ValueError, "empty separator");
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5948 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5949 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5950 else
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5951 return rsplit_substring(self,list,substring,maxcount);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5952 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5953
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
5954 static
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5955 PyObject *replace(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5956 PyUnicodeObject *str1,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5957 PyUnicodeObject *str2,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
5958 Py_ssize_t maxcount)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5959 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5960 PyUnicodeObject *u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5961
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5962 if (maxcount < 0)
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5963 maxcount = PY_SSIZE_T_MAX;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5964
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5965 if (str1->length == str2->length) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5966 /* same length */
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
5967 Py_ssize_t i;
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5968 if (str1->length == 1) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5969 /* replace characters */
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5970 Py_UNICODE u1, u2;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5971 if (!findchar(self->str, self->length, str1->str[0]))
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5972 goto nothing;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5973 u = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, self->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5974 if (!u)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5975 return NULL;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5976 Py_UNICODE_COPY(u->str, self->str, self->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5977 u1 = str1->str[0];
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5978 u2 = str2->str[0];
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5979 for (i = 0; i < u->length; i++)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5980 if (u->str[i] == u1) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5981 if (--maxcount < 0)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5982 break;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5983 u->str[i] = u2;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5984 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5985 } else {
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5986 i = fastsearch(
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5987 self->str, self->length, str1->str, str1->length, FAST_SEARCH
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5988 );
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5989 if (i < 0)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5990 goto nothing;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5991 u = (PyUnicodeObject*) PyUnicode_FromUnicode(NULL, self->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5992 if (!u)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5993 return NULL;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5994 Py_UNICODE_COPY(u->str, self->str, self->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5995 while (i <= self->length - str1->length)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5996 if (Py_UNICODE_MATCH(self, i, str1)) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5997 if (--maxcount < 0)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5998 break;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
5999 Py_UNICODE_COPY(u->str+i, str2->str, str2->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6000 i += str1->length;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6001 } else
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6002 i++;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6003 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6004 } else {
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6005
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6006 Py_ssize_t n, i, j, e;
38299
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6007 Py_ssize_t product, new_size, delta;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6008 Py_UNICODE *p;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6009
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6010 /* replace strings */
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6011 n = stringlib_count(self->str, self->length, str1->str, str1->length);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6012 if (n > maxcount)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6013 n = maxcount;
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6014 if (n == 0)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6015 goto nothing;
38299
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6016 /* new_size = self->length + n * (str2->length - str1->length)); */
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6017 delta = (str2->length - str1->length);
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6018 if (delta == 0) {
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6019 new_size = self->length;
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6020 } else {
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6021 product = n * (str2->length - str1->length);
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6022 if ((product / (str2->length - str1->length)) != n) {
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6023 PyErr_SetString(PyExc_OverflowError,
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6024 "replace string is too long");
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6025 return NULL;
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6026 }
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6027 new_size = self->length + product;
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6028 if (new_size < 0) {
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6029 PyErr_SetString(PyExc_OverflowError,
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6030 "replace string is too long");
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6031 return NULL;
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6032 }
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6033 }
ae6bbf2f2a41 needforspeed: check for overflow in replace (from Andrew Dalke)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38278
diff changeset
6034 u = _PyUnicode_New(new_size);
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6035 if (!u)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6036 return NULL;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6037 i = 0;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6038 p = u->str;
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6039 e = self->length - str1->length;
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6040 if (str1->length > 0) {
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6041 while (n-- > 0) {
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6042 /* look for next match */
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6043 j = i;
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6044 while (j <= e) {
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6045 if (Py_UNICODE_MATCH(self, j, str1))
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6046 break;
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6047 j++;
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6048 }
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6049 if (j > i) {
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6050 if (j > e)
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6051 break;
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6052 /* copy unchanged part [i:j] */
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6053 Py_UNICODE_COPY(p, self->str+i, j-i);
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6054 p += j - i;
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6055 }
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6056 /* copy substitution string */
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6057 if (str2->length > 0) {
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6058 Py_UNICODE_COPY(p, str2->str, str2->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6059 p += str2->length;
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6060 }
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6061 i = j + str1->length;
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6062 }
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6063 if (i < self->length)
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6064 /* copy tail [i:] */
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6065 Py_UNICODE_COPY(p, self->str+i, self->length-i);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6066 } else {
38440
5b0753b9d16a needforspeed: replace improvements, changed to Py_LOCAL_INLINE
Fredrik Lundh <fredrik@pythonware.com>
parents: 38427
diff changeset
6067 /* interleave */
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6068 while (n > 0) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6069 Py_UNICODE_COPY(p, str2->str, str2->length);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6070 p += str2->length;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6071 if (--n <= 0)
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6072 break;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6073 *p++ = self->str[i++];
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6074 }
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6075 Py_UNICODE_COPY(p, self->str+i, self->length-i);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6076 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6077 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6078 return (PyObject *) u;
38278
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6079
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6080 nothing:
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6081 /* nothing to replace; return original string (when possible) */
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6082 if (PyUnicode_CheckExact(self)) {
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6083 Py_INCREF(self);
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6084 return (PyObject *) self;
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6085 }
260d5a6ce30f needforspeed: refactored the replace code slightly; special-case
Fredrik Lundh <fredrik@pythonware.com>
parents: 38276
diff changeset
6086 return PyUnicode_FromUnicode(self->str, self->length);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6087 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6088
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6089 /* --- Unicode Object Methods --------------------------------------------- */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6090
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6091 PyDoc_STRVAR(title__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6092 "S.title() -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6093 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6094 Return a titlecased version of S, i.e. words start with title case\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6095 characters, all remaining cased characters have lower case.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6096
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6097 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6098 unicode_title(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6099 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6100 return fixup(self, fixtitle);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6101 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6102
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6103 PyDoc_STRVAR(capitalize__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6104 "S.capitalize() -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6105 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6106 Return a capitalized version of S, i.e. make the first character\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6107 have upper case.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6108
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6109 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6110 unicode_capitalize(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6111 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6112 return fixup(self, fixcapitalize);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6113 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6114
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6115 #if 0
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6116 PyDoc_STRVAR(capwords__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6117 "S.capwords() -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6118 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6119 Apply .capitalize() to all words in S and return the result with\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6120 normalized whitespace (all whitespace strings are replaced by ' ').");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6121
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6122 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6123 unicode_capwords(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6124 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6125 PyObject *list;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6126 PyObject *item;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6127 Py_ssize_t i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6128
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6129 /* Split into words */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6130 list = split(self, NULL, -1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6131 if (!list)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6132 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6133
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6134 /* Capitalize each word */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6135 for (i = 0; i < PyList_GET_SIZE(list); i++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6136 item = fixup((PyUnicodeObject *)PyList_GET_ITEM(list, i),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6137 fixcapitalize);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6138 if (item == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6139 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6140 Py_DECREF(PyList_GET_ITEM(list, i));
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6141 PyList_SET_ITEM(list, i, item);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6142 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6143
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6144 /* Join the words to form a new string */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6145 item = PyUnicode_Join(NULL, list);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6146
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6147 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6148 Py_DECREF(list);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6149 return (PyObject *)item;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6150 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6151 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6152
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6153 /* Argument converter. Coerces to a single unicode character */
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6154
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6155 static int
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6156 convert_uc(PyObject *obj, void *addr)
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6157 {
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6158 Py_UNICODE *fillcharloc = (Py_UNICODE *)addr;
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6159 PyObject *uniobj;
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6160 Py_UNICODE *unistr;
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6161
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6162 uniobj = PyUnicode_FromObject(obj);
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6163 if (uniobj == NULL) {
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6164 PyErr_SetString(PyExc_TypeError,
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6165 "The fill character cannot be converted to Unicode");
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6166 return 0;
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6167 }
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6168 if (PyUnicode_GET_SIZE(uniobj) != 1) {
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6169 PyErr_SetString(PyExc_TypeError,
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6170 "The fill character must be exactly one character long");
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6171 Py_DECREF(uniobj);
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6172 return 0;
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6173 }
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6174 unistr = PyUnicode_AS_UNICODE(uniobj);
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6175 *fillcharloc = unistr[0];
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6176 Py_DECREF(uniobj);
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6177 return 1;
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6178 }
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6179
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6180 PyDoc_STRVAR(center__doc__,
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6181 "S.center(width[, fillchar]) -> unicode\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6182 \n\
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6183 Return S centered in a Unicode string of length width. Padding is\n\
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6184 done using the specified fill character (default is a space)");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6185
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6186 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6187 unicode_center(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6188 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6189 Py_ssize_t marg, left;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6190 Py_ssize_t width;
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6191 Py_UNICODE fillchar = ' ';
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6192
36532
0c096be27000 Use correct PyArg_Parse format char for Py_ssize_t in unicode.center().
Thomas Wouters <thomas@python.org>
parents: 36519
diff changeset
6193 if (!PyArg_ParseTuple(args, "n|O&:center", &width, convert_uc, &fillchar))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6194 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6195
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
6196 if (self->length >= width && PyUnicode_CheckExact(self)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6197 Py_INCREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6198 return (PyObject*) self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6199 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6200
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6201 marg = width - self->length;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6202 left = marg / 2 + (marg & width & 1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6203
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
6204 return (PyObject*) pad(self, left, marg - left, fillchar);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6205 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6206
13890
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6207 #if 0
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6208
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6209 /* This code should go into some future Unicode collation support
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6210 module. The basic comparison should compare ordinals on a naive
13946
7a99450da2d6 Clean up warning from Monterey compiler.
Trent Mick <trentm@activestate.com>
parents: 13890
diff changeset
6211 basis (this is what Java does and thus JPython too). */
13890
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6212
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6213 /* speedy UTF-16 code point order comparison */
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6214 /* gleaned from: */
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6215 /* http://www-4.ibm.com/software/developer/library/utf16.html?dwzone=unicode */
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6216
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
6217 static short utf16Fixup[32] =
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6218 {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6219 0, 0, 0, 0, 0, 0, 0, 0,
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6220 0, 0, 0, 0, 0, 0, 0, 0,
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6221 0, 0, 0, 0, 0, 0, 0, 0,
13411
9a8391054840 New surrogate support in the UTF-8 codec. By Bill Tutt.
Marc-André Lemburg <mal@egenix.com>
parents: 13401
diff changeset
6222 0, 0, 0, 0x2000, -0x800, -0x800, -0x800, -0x800
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6223 };
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6224
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6225 static int
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6226 unicode_compare(PyUnicodeObject *str1, PyUnicodeObject *str2)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6227 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6228 Py_ssize_t len1, len2;
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6229
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6230 Py_UNICODE *s1 = str1->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6231 Py_UNICODE *s2 = str2->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6232
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6233 len1 = str1->length;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6234 len2 = str2->length;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6235
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6236 while (len1 > 0 && len2 > 0) {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6237 Py_UNICODE c1, c2;
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6238
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6239 c1 = *s1++;
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6240 c2 = *s2++;
18279
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6241
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6242 if (c1 > (1<<11) * 26)
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6243 c1 += utf16Fixup[c1>>11];
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6244 if (c2 > (1<<11) * 26)
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6245 c2 += utf16Fixup[c2>>11];
18279
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6246 /* now c1 and c2 are in UTF-32-compatible order */
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6247
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6248 if (c1 != c2)
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6249 return (c1 < c2) ? -1 : 1;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6250
13337
e101600adee4 Bill Tutt:
Marc-André Lemburg <mal@egenix.com>
parents: 13209
diff changeset
6251 len1--; len2--;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6252 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6253
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6254 return (len1 < len2) ? -1 : (len1 != len2);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6255 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6256
13890
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6257 #else
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6258
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6259 static int
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6260 unicode_compare(PyUnicodeObject *str1, PyUnicodeObject *str2)
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6261 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6262 register Py_ssize_t len1, len2;
13890
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6263
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6264 Py_UNICODE *s1 = str1->str;
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6265 Py_UNICODE *s2 = str2->str;
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6266
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6267 len1 = str1->length;
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6268 len2 = str2->length;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6269
13890
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6270 while (len1 > 0 && len2 > 0) {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6271 Py_UNICODE c1, c2;
18279
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6272
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6273 c1 = *s1++;
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6274 c2 = *s2++;
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6275
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6276 if (c1 != c2)
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6277 return (c1 < c2) ? -1 : 1;
8b7ff44b4669 experimental UCS-4 support: made compare a bit more robust, in case
Fredrik Lundh <fredrik@pythonware.com>
parents: 18278
diff changeset
6278
13890
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6279 len1--; len2--;
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6280 }
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6281
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6282 return (len1 < len2) ? -1 : (len1 != len2);
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6283 }
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6284
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6285 #endif
8f0fc0b5f784 Removing UTF-16 aware Unicode comparison code. This kind of compare
Marc-André Lemburg <mal@egenix.com>
parents: 13837
diff changeset
6286
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6287 int PyUnicode_Compare(PyObject *left,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6288 PyObject *right)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6289 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6290 PyUnicodeObject *u = NULL, *v = NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6291 int result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6292
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6293 /* Coerce the two arguments */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6294 u = (PyUnicodeObject *)PyUnicode_FromObject(left);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6295 if (u == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6296 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6297 v = (PyUnicodeObject *)PyUnicode_FromObject(right);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6298 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6299 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6300
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 13576
diff changeset
6301 /* Shortcut for empty or interned objects */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6302 if (v == u) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6303 Py_DECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6304 Py_DECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6305 return 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6306 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6307
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6308 result = unicode_compare(u, v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6309
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6310 Py_DECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6311 Py_DECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6312 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6313
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6314 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6315 Py_XDECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6316 Py_XDECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6317 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6318 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6319
39487
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6320 PyObject *PyUnicode_RichCompare(PyObject *left,
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6321 PyObject *right,
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6322 int op)
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6323 {
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6324 int result;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6325
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6326 result = PyUnicode_Compare(left, right);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6327 if (result == -1 && PyErr_Occurred())
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6328 goto onError;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6329
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6330 /* Convert the return value to a Boolean */
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6331 switch (op) {
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6332 case Py_EQ:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6333 result = (result == 0);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6334 break;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6335 case Py_NE:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6336 result = (result != 0);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6337 break;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6338 case Py_LE:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6339 result = (result <= 0);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6340 break;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6341 case Py_GE:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6342 result = (result >= 0);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6343 break;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6344 case Py_LT:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6345 result = (result == -1);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6346 break;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6347 case Py_GT:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6348 result = (result == 1);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6349 break;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6350 }
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6351 return PyBool_FromLong(result);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6352
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6353 onError:
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6354
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6355 /* Standard case
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6356
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6357 Type errors mean that PyUnicode_FromObject() could not convert
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6358 one of the arguments (usually the right hand side) to Unicode,
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6359 ie. we can't handle the comparison request. However, it is
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6360 possible that the other object knows a comparison method, which
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6361 is why we return Py_NotImplemented to give the other object a
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6362 chance.
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6363
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6364 */
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6365 if (PyErr_ExceptionMatches(PyExc_TypeError)) {
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6366 PyErr_Clear();
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6367 Py_INCREF(Py_NotImplemented);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6368 return Py_NotImplemented;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6369 }
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6370 if (op != Py_EQ && op != Py_NE)
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6371 return NULL;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6372
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6373 /* Equality comparison.
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6374
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6375 This is a special case: we silence any PyExc_UnicodeDecodeError
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6376 and instead turn it into a PyErr_UnicodeWarning.
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6377
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6378 */
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6379 if (!PyErr_ExceptionMatches(PyExc_UnicodeDecodeError))
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6380 return NULL;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6381 PyErr_Clear();
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6382 if (PyErr_Warn(PyExc_UnicodeWarning,
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6383 (op == Py_EQ) ?
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6384 "Unicode equal comparison "
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6385 "failed to convert both arguments to Unicode - "
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6386 "interpreting them as being unequal" :
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6387 "Unicode unequal comparison "
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6388 "failed to convert both arguments to Unicode - "
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6389 "interpreting them as being unequal"
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6390 ) < 0)
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6391 return NULL;
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6392 result = (op == Py_NE);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6393 return PyBool_FromLong(result);
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6394 }
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
6395
11997
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6396 int PyUnicode_Contains(PyObject *container,
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6397 PyObject *element)
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6398 {
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6399 PyObject *str, *sub;
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6400 int result;
11997
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6401
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6402 /* Coerce the two arguments */
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6403 sub = PyUnicode_FromObject(element);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6404 if (!sub) {
13049
0be4b7b650e3 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12957
diff changeset
6405 PyErr_SetString(PyExc_TypeError,
24598
12e146171f3d Committing patch #591250 which provides "str1 in str2" when str1 is a
Barry Warsaw <barry@python.org>
parents: 24392
diff changeset
6406 "'in <string>' requires string as left operand");
38243
0ba711a496e6 needforspeed: fixed unicode "in" operator to use same implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38241
diff changeset
6407 return -1;
0ba711a496e6 needforspeed: fixed unicode "in" operator to use same implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38241
diff changeset
6408 }
0ba711a496e6 needforspeed: fixed unicode "in" operator to use same implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38241
diff changeset
6409
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6410 str = PyUnicode_FromObject(container);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6411 if (!str) {
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6412 Py_DECREF(sub);
38243
0ba711a496e6 needforspeed: fixed unicode "in" operator to use same implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38241
diff changeset
6413 return -1;
0ba711a496e6 needforspeed: fixed unicode "in" operator to use same implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38241
diff changeset
6414 }
11997
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6415
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6416 result = stringlib_contains_obj(str, sub);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6417
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6418 Py_DECREF(str);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6419 Py_DECREF(sub);
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6420
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6421 return result;
11997
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6422 }
38310eb0659e Add sq_contains implementation.
Guido van Rossum <guido@python.org>
parents: 11968
diff changeset
6423
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6424 /* Concat to string or Unicode object giving a new Unicode object. */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6425
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6426 PyObject *PyUnicode_Concat(PyObject *left,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6427 PyObject *right)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6428 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6429 PyUnicodeObject *u = NULL, *v = NULL, *w;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6430
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6431 /* Coerce the two arguments */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6432 u = (PyUnicodeObject *)PyUnicode_FromObject(left);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6433 if (u == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6434 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6435 v = (PyUnicodeObject *)PyUnicode_FromObject(right);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6436 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6437 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6438
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6439 /* Shortcuts */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6440 if (v == unicode_empty) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6441 Py_DECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6442 return (PyObject *)u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6443 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6444 if (u == unicode_empty) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6445 Py_DECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6446 return (PyObject *)v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6447 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6448
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6449 /* Concat the two Unicode strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6450 w = _PyUnicode_New(u->length + v->length);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6451 if (w == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6452 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6453 Py_UNICODE_COPY(w->str, u->str, u->length);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6454 Py_UNICODE_COPY(w->str + u->length, v->str, v->length);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6455
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6456 Py_DECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6457 Py_DECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6458 return (PyObject *)w;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6459
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6460 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6461 Py_XDECREF(u);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6462 Py_XDECREF(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6463 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6464 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6465
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6466 PyDoc_STRVAR(count__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6467 "S.count(sub[, start[, end]]) -> int\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6468 \n\
38236
7e16fcce34a1 docstring tweaks: count counts non-overlapping substrings, not
Fredrik Lundh <fredrik@pythonware.com>
parents: 38182
diff changeset
6469 Return the number of non-overlapping occurrences of substring sub in\n\
7e16fcce34a1 docstring tweaks: count counts non-overlapping substrings, not
Fredrik Lundh <fredrik@pythonware.com>
parents: 38182
diff changeset
6470 Unicode string S[start:end]. Optional arguments start and end are\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6471 interpreted as in slice notation.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6472
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6473 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6474 unicode_count(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6475 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6476 PyUnicodeObject *substring;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6477 Py_ssize_t start = 0;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
6478 Py_ssize_t end = PY_SSIZE_T_MAX;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6479 PyObject *result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6480
12629
3efeaa7521e4 Trent Mick:
Guido van Rossum <guido@python.org>
parents: 12582
diff changeset
6481 if (!PyArg_ParseTuple(args, "O|O&O&:count", &substring,
3efeaa7521e4 Trent Mick:
Guido van Rossum <guido@python.org>
parents: 12582
diff changeset
6482 _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6483 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6484
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6485 substring = (PyUnicodeObject *)PyUnicode_FromObject(
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6486 (PyObject *)substring);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6487 if (substring == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6488 return NULL;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6489
38402
6b03a47eb54d needforspeed: use a macro to fix slice indexes
Fredrik Lundh <fredrik@pythonware.com>
parents: 38401
diff changeset
6490 FIX_START_END(self);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6491
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6492 result = PyInt_FromSsize_t(
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6493 stringlib_count(self->str + start, end - start,
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6494 substring->str, substring->length)
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6495 );
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6496
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6497 Py_DECREF(substring);
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
6498
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6499 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6500 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6501
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6502 PyDoc_STRVAR(encode__doc__,
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6503 "S.encode([encoding[,errors]]) -> string or unicode\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6504 \n\
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6505 Encodes S using the codec registered for encoding. encoding defaults\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6506 to the default encoding. errors may be given to set a different error\n\
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
6507 handling scheme. Default is 'strict' meaning that encoding errors raise\n\
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
6508 a UnicodeEncodeError. Other possible values are 'ignore', 'replace' and\n\
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
6509 'xmlcharrefreplace' as well as any other name registered with\n\
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 25005
diff changeset
6510 codecs.register_error that can handle UnicodeEncodeErrors.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6511
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6512 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6513 unicode_encode(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6514 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6515 char *encoding = NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6516 char *errors = NULL;
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6517 PyObject *v;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6518
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6519 if (!PyArg_ParseTuple(args, "|ss:encode", &encoding, &errors))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6520 return NULL;
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6521 v = PyUnicode_AsEncodedObject((PyObject *)self, encoding, errors);
32552
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6522 if (v == NULL)
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6523 goto onError;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
6524 if (!PyString_Check(v) && !PyUnicode_Check(v)) {
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6525 PyErr_Format(PyExc_TypeError,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6526 "encoder did not return a string/unicode object "
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6527 "(type=%.400s)",
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
6528 Py_TYPE(v)->tp_name);
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6529 Py_DECREF(v);
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6530 return NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6531 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6532 return v;
32552
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6533
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6534 onError:
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6535 return NULL;
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6536 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6537
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6538 PyDoc_STRVAR(decode__doc__,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6539 "S.decode([encoding[,errors]]) -> string or unicode\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6540 \n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6541 Decodes S using the codec registered for encoding. encoding defaults\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6542 to the default encoding. errors may be given to set a different error\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6543 handling scheme. Default is 'strict' meaning that encoding errors raise\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6544 a UnicodeDecodeError. Other possible values are 'ignore' and 'replace'\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6545 as well as any other name registerd with codecs.register_error that is\n\
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6546 able to handle UnicodeDecodeErrors.");
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6547
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6548 static PyObject *
32571
bec0c7618625 Fix a copy&paste typo.
Marc-André Lemburg <mal@egenix.com>
parents: 32552
diff changeset
6549 unicode_decode(PyUnicodeObject *self, PyObject *args)
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6550 {
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6551 char *encoding = NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6552 char *errors = NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6553 PyObject *v;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6554
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6555 if (!PyArg_ParseTuple(args, "|ss:decode", &encoding, &errors))
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6556 return NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6557 v = PyUnicode_AsDecodedObject((PyObject *)self, encoding, errors);
32552
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6558 if (v == NULL)
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6559 goto onError;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
6560 if (!PyString_Check(v) && !PyUnicode_Check(v)) {
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6561 PyErr_Format(PyExc_TypeError,
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6562 "decoder did not return a string/unicode object "
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6563 "(type=%.400s)",
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
6564 Py_TYPE(v)->tp_name);
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6565 Py_DECREF(v);
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6566 return NULL;
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6567 }
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
6568 return v;
32552
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6569
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6570 onError:
3706a3704aea .encode()/.decode() patch part 2.
Marc-André Lemburg <mal@egenix.com>
parents: 32551
diff changeset
6571 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6572 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6573
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6574 PyDoc_STRVAR(expandtabs__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6575 "S.expandtabs([tabsize]) -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6576 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6577 Return a copy of S where all tab characters are expanded using spaces.\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6578 If tabsize is not given, a tab size of 8 characters is assumed.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6579
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6580 static PyObject*
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6581 unicode_expandtabs(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6582 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6583 Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6584 Py_UNICODE *p;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6585 Py_UNICODE *q;
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6586 Py_UNICODE *qe;
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6587 Py_ssize_t i, j, incr;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6588 PyUnicodeObject *u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6589 int tabsize = 8;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6590
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6591 if (!PyArg_ParseTuple(args, "|i:expandtabs", &tabsize))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6592 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6593
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 13576
diff changeset
6594 /* First pass: determine size of output string */
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6595 i = 0; /* chars up to and including most recent \n or \r */
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6596 j = 0; /* chars since most recent \n or \r (use in tab calculations) */
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6597 e = self->str + self->length; /* end of input */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6598 for (p = self->str; p < e; p++)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6599 if (*p == '\t') {
42004
e62b606eff0b Prevent expandtabs() on string and unicode objects from causing a segfault when
Neal Norwitz <nnorwitz@gmail.com>
parents: 40908
diff changeset
6600 if (tabsize > 0) {
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6601 incr = tabsize - (j % tabsize); /* cannot overflow */
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6602 if (j > PY_SSIZE_T_MAX - incr)
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6603 goto overflow1;
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6604 j += incr;
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6605 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6606 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6607 else {
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6608 if (j > PY_SSIZE_T_MAX - 1)
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6609 goto overflow1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6610 j++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6611 if (*p == '\n' || *p == '\r') {
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6612 if (i > PY_SSIZE_T_MAX - j)
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6613 goto overflow1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6614 i += j;
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6615 j = 0;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6616 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6617 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6618
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6619 if (i > PY_SSIZE_T_MAX - j)
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6620 goto overflow1;
42004
e62b606eff0b Prevent expandtabs() on string and unicode objects from causing a segfault when
Neal Norwitz <nnorwitz@gmail.com>
parents: 40908
diff changeset
6621
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6622 /* Second pass: create output string and fill it */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6623 u = _PyUnicode_New(i + j);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6624 if (!u)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6625 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6626
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6627 j = 0; /* same as in first pass */
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6628 q = u->str; /* next output char */
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6629 qe = u->str + u->length; /* end of output */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6630
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6631 for (p = self->str; p < e; p++)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6632 if (*p == '\t') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6633 if (tabsize > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6634 i = tabsize - (j % tabsize);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6635 j += i;
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6636 while (i--) {
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6637 if (q >= qe)
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6638 goto overflow2;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6639 *q++ = ' ';
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6640 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6641 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6642 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6643 else {
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6644 if (q >= qe)
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6645 goto overflow2;
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6646 *q++ = *p;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6647 j++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6648 if (*p == '\n' || *p == '\r')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6649 j = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6650 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6651
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6652 return (PyObject*) u;
45731
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6653
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6654 overflow2:
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6655 Py_DECREF(u);
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6656 overflow1:
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6657 PyErr_SetString(PyExc_OverflowError, "new string is too long");
162c6952ea34 Fix the overflows in expandtabs(). "This time for sure!"
Guido van Rossum <guido@python.org>
parents: 45500
diff changeset
6658 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6659 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6660
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6661 PyDoc_STRVAR(find__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6662 "S.find(sub [,start [,end]]) -> int\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6663 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6664 Return the lowest index in S where substring sub is found,\n\
42402
251033a1c2e3 Bug #1763149: use proper slice syntax in docstring.
Georg Brandl <georg@python.org>
parents: 42333
diff changeset
6665 such that sub is contained within s[start:end]. Optional\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6666 arguments start and end are interpreted as in slice notation.\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6667 \n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6668 Return -1 on failure.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6669
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6670 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6671 unicode_find(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6672 {
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6673 PyObject *substring;
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6674 Py_ssize_t start;
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6675 Py_ssize_t end;
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6676 Py_ssize_t result;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6677
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6678 if (!_ParseTupleFinds(args, &substring, &start, &end))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6679 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6680
38441
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6681 result = stringlib_find_slice(
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6682 PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6683 PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6684 start, end
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6685 );
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6686
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6687 Py_DECREF(substring);
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6688
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6689 return PyInt_FromSsize_t(result);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6690 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6691
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6692 static PyObject *
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6693 unicode_getitem(PyUnicodeObject *self, Py_ssize_t index)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6694 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6695 if (index < 0 || index >= self->length) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6696 PyErr_SetString(PyExc_IndexError, "string index out of range");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6697 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6698 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6699
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6700 return (PyObject*) PyUnicode_FromUnicode(&self->str[index], 1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6701 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6702
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6703 static long
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6704 unicode_hash(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6705 {
13522
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6706 /* Since Unicode objects compare equal to their ASCII string
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6707 counterparts, they should use the individual character values
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6708 as basis for their hash value. This is needed to assure that
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6709 strings and Unicode objects behave in the same way as
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6710 dictionary keys. */
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6711
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6712 register Py_ssize_t len;
13522
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6713 register Py_UNICODE *p;
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6714 register long x;
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6715
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6716 if (self->hash != -1)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6717 return self->hash;
13522
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6718 len = PyUnicode_GET_SIZE(self);
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6719 p = PyUnicode_AS_UNICODE(self);
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6720 x = *p << 7;
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6721 while (--len >= 0)
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6722 x = (1000003*x) ^ *p++;
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6723 x ^= PyUnicode_GET_SIZE(self);
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6724 if (x == -1)
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6725 x = -2;
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6726 self->hash = x;
acc4638f37d5 - changed hash calculation for unicode strings. the new
Fredrik Lundh <fredrik@pythonware.com>
parents: 13411
diff changeset
6727 return x;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6728 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6729
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6730 PyDoc_STRVAR(index__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6731 "S.index(sub [,start [,end]]) -> int\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6732 \n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6733 Like S.find() but raise ValueError when the substring is not found.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6734
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6735 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6736 unicode_index(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6737 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6738 Py_ssize_t result;
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6739 PyObject *substring;
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6740 Py_ssize_t start;
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6741 Py_ssize_t end;
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6742
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
6743 if (!_ParseTupleFinds(args, &substring, &start, &end))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6744 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6745
38441
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6746 result = stringlib_find_slice(
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6747 PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6748 PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6749 start, end
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
6750 );
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6751
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6752 Py_DECREF(substring);
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6753
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6754 if (result < 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6755 PyErr_SetString(PyExc_ValueError, "substring not found");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6756 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6757 }
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
6758
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
6759 return PyInt_FromSsize_t(result);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6760 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6761
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6762 PyDoc_STRVAR(islower__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6763 "S.islower() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6764 \n\
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6765 Return True if all cased characters in S are lowercase and there is\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6766 at least one cased character in S, False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6767
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6768 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6769 unicode_islower(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6770 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6771 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6772 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6773 int cased;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6774
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6775 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6776 if (PyUnicode_GET_SIZE(self) == 1)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6777 return PyBool_FromLong(Py_UNICODE_ISLOWER(*p));
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6778
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6779 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6780 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6781 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6782
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6783 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6784 cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6785 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6786 register const Py_UNICODE ch = *p;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6787
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6788 if (Py_UNICODE_ISUPPER(ch) || Py_UNICODE_ISTITLE(ch))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6789 return PyBool_FromLong(0);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6790 else if (!cased && Py_UNICODE_ISLOWER(ch))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6791 cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6792 }
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6793 return PyBool_FromLong(cased);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6794 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6795
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6796 PyDoc_STRVAR(isupper__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6797 "S.isupper() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6798 \n\
30707
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6799 Return True if all cased characters in S are uppercase and there is\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6800 at least one cased character in S, False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6801
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6802 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6803 unicode_isupper(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6804 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6805 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6806 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6807 int cased;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6808
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6809 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6810 if (PyUnicode_GET_SIZE(self) == 1)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6811 return PyBool_FromLong(Py_UNICODE_ISUPPER(*p) != 0);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6812
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6813 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6814 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6815 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6816
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6817 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6818 cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6819 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6820 register const Py_UNICODE ch = *p;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6821
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6822 if (Py_UNICODE_ISLOWER(ch) || Py_UNICODE_ISTITLE(ch))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6823 return PyBool_FromLong(0);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6824 else if (!cased && Py_UNICODE_ISUPPER(ch))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6825 cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6826 }
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6827 return PyBool_FromLong(cased);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6828 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6829
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6830 PyDoc_STRVAR(istitle__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6831 "S.istitle() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6832 \n\
30707
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6833 Return True if S is a titlecased string and there is at least one\n\
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6834 character in S, i.e. upper- and titlecase characters may only\n\
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6835 follow uncased characters and lowercase characters only cased ones.\n\
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6836 Return False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6837
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6838 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6839 unicode_istitle(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6840 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6841 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6842 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6843 int cased, previous_is_cased;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6844
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6845 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6846 if (PyUnicode_GET_SIZE(self) == 1)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6847 return PyBool_FromLong((Py_UNICODE_ISTITLE(*p) != 0) ||
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6848 (Py_UNICODE_ISUPPER(*p) != 0));
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6849
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6850 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6851 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6852 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6853
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6854 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6855 cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6856 previous_is_cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6857 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6858 register const Py_UNICODE ch = *p;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
6859
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6860 if (Py_UNICODE_ISUPPER(ch) || Py_UNICODE_ISTITLE(ch)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6861 if (previous_is_cased)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6862 return PyBool_FromLong(0);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6863 previous_is_cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6864 cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6865 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6866 else if (Py_UNICODE_ISLOWER(ch)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6867 if (!previous_is_cased)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6868 return PyBool_FromLong(0);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6869 previous_is_cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6870 cased = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6871 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6872 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6873 previous_is_cased = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6874 }
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6875 return PyBool_FromLong(cased);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6876 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6877
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6878 PyDoc_STRVAR(isspace__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6879 "S.isspace() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6880 \n\
30707
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6881 Return True if all characters in S are whitespace\n\
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6882 and there is at least one character in S, False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6883
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6884 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6885 unicode_isspace(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6886 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6887 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6888 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6889
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6890 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6891 if (PyUnicode_GET_SIZE(self) == 1 &&
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6892 Py_UNICODE_ISSPACE(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6893 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6894
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6895 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6896 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6897 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6898
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6899 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6900 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6901 if (!Py_UNICODE_ISSPACE(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6902 return PyBool_FromLong(0);
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6903 }
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6904 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6905 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6906
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6907 PyDoc_STRVAR(isalpha__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6908 "S.isalpha() -> bool\n\
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6909 \n\
30707
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6910 Return True if all characters in S are alphabetic\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6911 and there is at least one character in S, False otherwise.");
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6912
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6913 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6914 unicode_isalpha(PyUnicodeObject *self)
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6915 {
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6916 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6917 register const Py_UNICODE *e;
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6918
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6919 /* Shortcut for single character strings */
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6920 if (PyUnicode_GET_SIZE(self) == 1 &&
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6921 Py_UNICODE_ISALPHA(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6922 return PyBool_FromLong(1);
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6923
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6924 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6925 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6926 return PyBool_FromLong(0);
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6927
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6928 e = p + PyUnicode_GET_SIZE(self);
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6929 for (; p < e; p++) {
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6930 if (!Py_UNICODE_ISALPHA(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6931 return PyBool_FromLong(0);
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6932 }
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6933 return PyBool_FromLong(1);
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6934 }
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6935
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6936 PyDoc_STRVAR(isalnum__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6937 "S.isalnum() -> bool\n\
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6938 \n\
30707
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6939 Return True if all characters in S are alphanumeric\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6940 and there is at least one character in S, False otherwise.");
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6941
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6942 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6943 unicode_isalnum(PyUnicodeObject *self)
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6944 {
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6945 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6946 register const Py_UNICODE *e;
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6947
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6948 /* Shortcut for single character strings */
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6949 if (PyUnicode_GET_SIZE(self) == 1 &&
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6950 Py_UNICODE_ISALNUM(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6951 return PyBool_FromLong(1);
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6952
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6953 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6954 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6955 return PyBool_FromLong(0);
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6956
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6957 e = p + PyUnicode_GET_SIZE(self);
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6958 for (; p < e; p++) {
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6959 if (!Py_UNICODE_ISALNUM(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6960 return PyBool_FromLong(0);
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6961 }
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6962 return PyBool_FromLong(1);
13356
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6963 }
9a1b961e0c3a Added new .isalpha() and .isalnum() methods which provide interfaces
Marc-André Lemburg <mal@egenix.com>
parents: 13337
diff changeset
6964
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6965 PyDoc_STRVAR(isdecimal__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6966 "S.isdecimal() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6967 \n\
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6968 Return True if there are only decimal characters in S,\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6969 False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6970
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6971 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
6972 unicode_isdecimal(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6973 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6974 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6975 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6976
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6977 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6978 if (PyUnicode_GET_SIZE(self) == 1 &&
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6979 Py_UNICODE_ISDECIMAL(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6980 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6981
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6982 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
6983 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6984 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
6985
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6986 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6987 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6988 if (!Py_UNICODE_ISDECIMAL(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6989 return PyBool_FromLong(0);
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6990 }
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6991 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6992 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6993
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
6994 PyDoc_STRVAR(isdigit__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
6995 "S.isdigit() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6996 \n\
30707
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6997 Return True if all characters in S are digits\n\
9e21e56bf5f3 Patch #825679: Clarify semantics of .isfoo on empty strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 30653
diff changeset
6998 and there is at least one character in S, False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6999
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7000 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7001 unicode_isdigit(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7002 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7003 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7004 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7005
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7006 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7007 if (PyUnicode_GET_SIZE(self) == 1 &&
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7008 Py_UNICODE_ISDIGIT(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7009 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7010
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
7011 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
7012 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7013 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
7014
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7015 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7016 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7017 if (!Py_UNICODE_ISDIGIT(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7018 return PyBool_FromLong(0);
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7019 }
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7020 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7021 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7022
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7023 PyDoc_STRVAR(isnumeric__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7024 "S.isnumeric() -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7025 \n\
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7026 Return True if there are only numeric characters in S,\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7027 False otherwise.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7028
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7029 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7030 unicode_isnumeric(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7031 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7032 register const Py_UNICODE *p = PyUnicode_AS_UNICODE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7033 register const Py_UNICODE *e;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7034
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7035 /* Shortcut for single character strings */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7036 if (PyUnicode_GET_SIZE(self) == 1 &&
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7037 Py_UNICODE_ISNUMERIC(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7038 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7039
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
7040 /* Special case for empty strings */
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
7041 if (PyUnicode_GET_SIZE(self) == 0)
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7042 return PyBool_FromLong(0);
12924
2d0b99778edf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12909
diff changeset
7043
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7044 e = p + PyUnicode_GET_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7045 for (; p < e; p++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7046 if (!Py_UNICODE_ISNUMERIC(*p))
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7047 return PyBool_FromLong(0);
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7048 }
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7049 return PyBool_FromLong(1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7050 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7051
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7052 PyDoc_STRVAR(join__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7053 "S.join(sequence) -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7054 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7055 Return a string which is the concatenation of the strings in the\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7056 sequence. The separator between elements is S.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7057
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7058 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7059 unicode_join(PyObject *self, PyObject *data)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7060 {
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7061 return PyUnicode_Join(self, data);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7062 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7063
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7064 static Py_ssize_t
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7065 unicode_length(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7066 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7067 return self->length;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7068 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7069
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7070 PyDoc_STRVAR(ljust__doc__,
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31873
diff changeset
7071 "S.ljust(width[, fillchar]) -> int\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7072 \n\
49431
2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49386
diff changeset
7073 Return S left-justified in a Unicode string of length width. Padding is\n\
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7074 done using the specified fill character (default is a space).");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7075
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7076 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7077 unicode_ljust(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7078 {
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
7079 Py_ssize_t width;
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7080 Py_UNICODE fillchar = ' ';
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7081
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
7082 if (!PyArg_ParseTuple(args, "n|O&:ljust", &width, convert_uc, &fillchar))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7083 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7084
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
7085 if (self->length >= width && PyUnicode_CheckExact(self)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7086 Py_INCREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7087 return (PyObject*) self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7088 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7089
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7090 return (PyObject*) pad(self, 0, width - self->length, fillchar);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7091 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7092
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7093 PyDoc_STRVAR(lower__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7094 "S.lower() -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7095 \n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7096 Return a copy of the string S converted to lowercase.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7097
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7098 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7099 unicode_lower(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7100 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7101 return fixup(self, fixlower);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7102 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7103
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7104 #define LEFTSTRIP 0
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7105 #define RIGHTSTRIP 1
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7106 #define BOTHSTRIP 2
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7107
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7108 /* Arrays indexed by above */
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7109 static const char *stripformat[] = {"|O:lstrip", "|O:rstrip", "|O:strip"};
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7110
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7111 #define STRIPNAME(i) (stripformat[i]+3)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7112
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7113 /* externally visible for str.strip(unicode) */
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7114 PyObject *
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7115 _PyUnicode_XStrip(PyUnicodeObject *self, int striptype, PyObject *sepobj)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7116 {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7117 Py_UNICODE *s = PyUnicode_AS_UNICODE(self);
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7118 Py_ssize_t len = PyUnicode_GET_SIZE(self);
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7119 Py_UNICODE *sep = PyUnicode_AS_UNICODE(sepobj);
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7120 Py_ssize_t seplen = PyUnicode_GET_SIZE(sepobj);
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7121 Py_ssize_t i, j;
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7122
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7123 BLOOM_MASK sepmask = make_bloom_mask(sep, seplen);
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7124
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7125 i = 0;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7126 if (striptype != RIGHTSTRIP) {
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7127 while (i < len && BLOOM_MEMBER(sepmask, s[i], sep, seplen)) {
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7128 i++;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7129 }
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7130 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7131
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7132 j = len;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7133 if (striptype != LEFTSTRIP) {
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7134 do {
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7135 j--;
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7136 } while (j >= i && BLOOM_MEMBER(sepmask, s[j], sep, seplen));
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7137 j++;
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7138 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7139
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7140 if (i == 0 && j == len && PyUnicode_CheckExact(self)) {
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7141 Py_INCREF(self);
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7142 return (PyObject*)self;
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7143 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7144 else
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
7145 return PyUnicode_FromUnicode(s+i, j-i);
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7146 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7147
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7148
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7149 static PyObject *
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7150 do_strip(PyUnicodeObject *self, int striptype)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7151 {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7152 Py_UNICODE *s = PyUnicode_AS_UNICODE(self);
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7153 Py_ssize_t len = PyUnicode_GET_SIZE(self), i, j;
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7154
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7155 i = 0;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7156 if (striptype != RIGHTSTRIP) {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7157 while (i < len && Py_UNICODE_ISSPACE(s[i])) {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7158 i++;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7159 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7160 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7161
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7162 j = len;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7163 if (striptype != LEFTSTRIP) {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7164 do {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7165 j--;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7166 } while (j >= i && Py_UNICODE_ISSPACE(s[j]));
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7167 j++;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7168 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7169
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7170 if (i == 0 && j == len && PyUnicode_CheckExact(self)) {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7171 Py_INCREF(self);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7172 return (PyObject*)self;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7173 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7174 else
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7175 return PyUnicode_FromUnicode(s+i, j-i);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7176 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7177
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7178
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7179 static PyObject *
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7180 do_argstrip(PyUnicodeObject *self, int striptype, PyObject *args)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7181 {
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7182 PyObject *sep = NULL;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7183
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7184 if (!PyArg_ParseTuple(args, (char *)stripformat[striptype], &sep))
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7185 return NULL;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7186
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7187 if (sep != NULL && sep != Py_None) {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7188 if (PyUnicode_Check(sep))
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7189 return _PyUnicode_XStrip(self, striptype, sep);
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
7190 else if (PyString_Check(sep)) {
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7191 PyObject *res;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7192 sep = PyUnicode_FromObject(sep);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7193 if (sep==NULL)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7194 return NULL;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7195 res = _PyUnicode_XStrip(self, striptype, sep);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7196 Py_DECREF(sep);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7197 return res;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7198 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7199 else {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7200 PyErr_Format(PyExc_TypeError,
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7201 "%s arg must be None, unicode or str",
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7202 STRIPNAME(striptype));
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7203 return NULL;
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7204 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7205 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7206
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7207 return do_strip(self, striptype);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7208 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7209
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7210
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7211 PyDoc_STRVAR(strip__doc__,
28575
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7212 "S.strip([chars]) -> unicode\n\
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7213 \n\
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7214 Return a copy of the string S with leading and trailing\n\
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7215 whitespace removed.\n\
28575
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7216 If chars is given and not None, remove characters in chars instead.\n\
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7217 If chars is a str, it will be converted to unicode before stripping");
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7218
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7219 static PyObject *
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7220 unicode_strip(PyUnicodeObject *self, PyObject *args)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7221 {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7222 if (PyTuple_GET_SIZE(args) == 0)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7223 return do_strip(self, BOTHSTRIP); /* Common case */
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7224 else
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7225 return do_argstrip(self, BOTHSTRIP, args);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7226 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7227
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7228
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7229 PyDoc_STRVAR(lstrip__doc__,
28575
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7230 "S.lstrip([chars]) -> unicode\n\
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7231 \n\
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7232 Return a copy of the string S with leading whitespace removed.\n\
28575
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7233 If chars is given and not None, remove characters in chars instead.\n\
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7234 If chars is a str, it will be converted to unicode before stripping");
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7235
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7236 static PyObject *
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7237 unicode_lstrip(PyUnicodeObject *self, PyObject *args)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7238 {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7239 if (PyTuple_GET_SIZE(args) == 0)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7240 return do_strip(self, LEFTSTRIP); /* Common case */
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7241 else
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7242 return do_argstrip(self, LEFTSTRIP, args);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7243 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7244
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7245
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7246 PyDoc_STRVAR(rstrip__doc__,
28575
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7247 "S.rstrip([chars]) -> unicode\n\
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7248 \n\
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7249 Return a copy of the string S with trailing whitespace removed.\n\
28575
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7250 If chars is given and not None, remove characters in chars instead.\n\
215f1f4851a4 Attempt to make all the various string *strip methods the same.
Neal Norwitz <nnorwitz@gmail.com>
parents: 28559
diff changeset
7251 If chars is a str, it will be converted to unicode before stripping");
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7252
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7253 static PyObject *
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7254 unicode_rstrip(PyUnicodeObject *self, PyObject *args)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7255 {
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7256 if (PyTuple_GET_SIZE(args) == 0)
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7257 return do_strip(self, RIGHTSTRIP); /* Common case */
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7258 else
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7259 return do_argstrip(self, RIGHTSTRIP, args);
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7260 }
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7261
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7262
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7263 static PyObject*
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7264 unicode_repeat(PyUnicodeObject *str, Py_ssize_t len)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7265 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7266 PyUnicodeObject *u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7267 Py_UNICODE *p;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7268 Py_ssize_t nchars;
14492
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7269 size_t nbytes;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7270
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7271 if (len < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7272 len = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7273
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
7274 if (len == 1 && PyUnicode_CheckExact(str)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7275 /* no repeat, return original string */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7276 Py_INCREF(str);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7277 return (PyObject*) str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7278 }
14492
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7279
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7280 /* ensure # of chars needed doesn't overflow int and # of bytes
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7281 * needed doesn't overflow size_t
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7282 */
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7283 nchars = len * str->length;
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7284 if (len && nchars / len != str->length) {
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7285 PyErr_SetString(PyExc_OverflowError,
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7286 "repeated string is too long");
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7287 return NULL;
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7288 }
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7289 nbytes = (nchars + 1) * sizeof(Py_UNICODE);
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7290 if (nbytes / sizeof(Py_UNICODE) != (size_t)(nchars + 1)) {
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7291 PyErr_SetString(PyExc_OverflowError,
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7292 "repeated string is too long");
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7293 return NULL;
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7294 }
7728b0ea9344 Fix for bug 113934. string*n and unicode*n did no overflow checking at
Tim Peters <tim.peters@gmail.com>
parents: 14388
diff changeset
7295 u = _PyUnicode_New(nchars);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7296 if (!u)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7297 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7298
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7299 p = u->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7300
38238
feb00df91922 needforspeed: speed up unicode repeat, unicode string copy
Fredrik Lundh <fredrik@pythonware.com>
parents: 38236
diff changeset
7301 if (str->length == 1 && len > 0) {
feb00df91922 needforspeed: speed up unicode repeat, unicode string copy
Fredrik Lundh <fredrik@pythonware.com>
parents: 38236
diff changeset
7302 Py_UNICODE_FILL(p, str->str[0], len);
38239
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7303 } else {
38241
5eaf8b9587d8 unicode_repeat(): Change type of local to Py_ssize_t,
Tim Peters <tim.peters@gmail.com>
parents: 38240
diff changeset
7304 Py_ssize_t done = 0; /* number of characters copied this far */
38239
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7305 if (done < nchars) {
38238
feb00df91922 needforspeed: speed up unicode repeat, unicode string copy
Fredrik Lundh <fredrik@pythonware.com>
parents: 38236
diff changeset
7306 Py_UNICODE_COPY(p, str->str, str->length);
38239
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7307 done = str->length;
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7308 }
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7309 while (done < nchars) {
46062
09adff3d6691 Try to fix a bunch of compiler warnings on Win64.
Neal Norwitz <nnorwitz@gmail.com>
parents: 46059
diff changeset
7310 Py_ssize_t n = (done <= nchars-done) ? done : nchars-done;
38239
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7311 Py_UNICODE_COPY(p+done, p, n);
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7312 done += n;
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7313 }
604af867c65f needforspeed: use memcpy for "long" strings; use a better algorithm
Fredrik Lundh <fredrik@pythonware.com>
parents: 38238
diff changeset
7314 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7315
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7316 return (PyObject*) u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7317 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7318
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7319 PyObject *PyUnicode_Replace(PyObject *obj,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7320 PyObject *subobj,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7321 PyObject *replobj,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7322 Py_ssize_t maxcount)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7323 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7324 PyObject *self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7325 PyObject *str1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7326 PyObject *str2;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7327 PyObject *result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7328
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7329 self = PyUnicode_FromObject(obj);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7330 if (self == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7331 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7332 str1 = PyUnicode_FromObject(subobj);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7333 if (str1 == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7334 Py_DECREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7335 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7336 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7337 str2 = PyUnicode_FromObject(replobj);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7338 if (str2 == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7339 Py_DECREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7340 Py_DECREF(str1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7341 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7342 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
7343 result = replace((PyUnicodeObject *)self,
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
7344 (PyUnicodeObject *)str1,
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
7345 (PyUnicodeObject *)str2,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7346 maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7347 Py_DECREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7348 Py_DECREF(str1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7349 Py_DECREF(str2);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7350 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7351 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7352
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7353 PyDoc_STRVAR(replace__doc__,
47388
d6639cd9575f #2999: fix name of third parameter in unicode.replace()'s docstring.
Georg Brandl <georg@python.org>
parents: 47294
diff changeset
7354 "S.replace (old, new[, count]) -> unicode\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7355 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7356 Return a copy of S with all occurrences of substring\n\
47388
d6639cd9575f #2999: fix name of third parameter in unicode.replace()'s docstring.
Georg Brandl <georg@python.org>
parents: 47294
diff changeset
7357 old replaced by new. If the optional argument count is\n\
d6639cd9575f #2999: fix name of third parameter in unicode.replace()'s docstring.
Georg Brandl <georg@python.org>
parents: 47294
diff changeset
7358 given, only the first count occurrences are replaced.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7359
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7360 static PyObject*
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7361 unicode_replace(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7362 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7363 PyUnicodeObject *str1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7364 PyUnicodeObject *str2;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7365 Py_ssize_t maxcount = -1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7366 PyObject *result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7367
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7368 if (!PyArg_ParseTuple(args, "OO|n:replace", &str1, &str2, &maxcount))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7369 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7370 str1 = (PyUnicodeObject *)PyUnicode_FromObject((PyObject *)str1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7371 if (str1 == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7372 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7373 str2 = (PyUnicodeObject *)PyUnicode_FromObject((PyObject *)str2);
27774
bfc524379307 Fix two refcounting bugs
Walter Dörwald <walter@livinglogic.de>
parents: 27555
diff changeset
7374 if (str2 == NULL) {
bfc524379307 Fix two refcounting bugs
Walter Dörwald <walter@livinglogic.de>
parents: 27555
diff changeset
7375 Py_DECREF(str1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7376 return NULL;
27774
bfc524379307 Fix two refcounting bugs
Walter Dörwald <walter@livinglogic.de>
parents: 27555
diff changeset
7377 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7378
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7379 result = replace(self, str1, str2, maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7380
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7381 Py_DECREF(str1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7382 Py_DECREF(str2);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7383 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7384 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7385
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7386 static
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7387 PyObject *unicode_repr(PyObject *unicode)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7388 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7389 return unicodeescape_string(PyUnicode_AS_UNICODE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7390 PyUnicode_GET_SIZE(unicode),
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7391 1);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7392 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7393
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7394 PyDoc_STRVAR(rfind__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7395 "S.rfind(sub [,start [,end]]) -> int\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7396 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7397 Return the highest index in S where substring sub is found,\n\
42402
251033a1c2e3 Bug #1763149: use proper slice syntax in docstring.
Georg Brandl <georg@python.org>
parents: 42333
diff changeset
7398 such that sub is contained within s[start:end]. Optional\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7399 arguments start and end are interpreted as in slice notation.\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7400 \n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7401 Return -1 on failure.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7402
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7403 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7404 unicode_rfind(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7405 {
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7406 PyObject *substring;
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7407 Py_ssize_t start;
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7408 Py_ssize_t end;
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7409 Py_ssize_t result;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7410
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7411 if (!_ParseTupleFinds(args, &substring, &start, &end))
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7412 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7413
38441
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7414 result = stringlib_rfind_slice(
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7415 PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7416 PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7417 start, end
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7418 );
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7419
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7420 Py_DECREF(substring);
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7421
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7422 return PyInt_FromSsize_t(result);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7423 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7424
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7425 PyDoc_STRVAR(rindex__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7426 "S.rindex(sub [,start [,end]]) -> int\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7427 \n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7428 Like S.rfind() but raise ValueError when the substring is not found.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7429
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7430 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7431 unicode_rindex(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7432 {
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7433 PyObject *substring;
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7434 Py_ssize_t start;
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7435 Py_ssize_t end;
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7436 Py_ssize_t result;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7437
43886
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7438 if (!_ParseTupleFinds(args, &substring, &start, &end))
25a5f6a30292 Now in find, rfind, index, and rindex, you can use None as defaults,
Facundo Batista <facundobatista@gmail.com>
parents: 43710
diff changeset
7439 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7440
38441
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7441 result = stringlib_rfind_slice(
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7442 PyUnicode_AS_UNICODE(self), PyUnicode_GET_SIZE(self),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7443 PyUnicode_AS_UNICODE(substring), PyUnicode_GET_SIZE(substring),
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7444 start, end
72327f8f4a79 needforspeed: stringlib refactoring: changed find_obj to find_slice,
Fredrik Lundh <fredrik@pythonware.com>
parents: 38440
diff changeset
7445 );
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7446
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7447 Py_DECREF(substring);
38424
3d81faf47af7 needforspeed: more stringlib refactoring
Fredrik Lundh <fredrik@pythonware.com>
parents: 38422
diff changeset
7448
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7449 if (result < 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7450 PyErr_SetString(PyExc_ValueError, "substring not found");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7451 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7452 }
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7453 return PyInt_FromSsize_t(result);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7454 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7455
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7456 PyDoc_STRVAR(rjust__doc__,
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7457 "S.rjust(width[, fillchar]) -> unicode\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7458 \n\
49431
2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49386
diff changeset
7459 Return S right-justified in a Unicode string of length width. Padding is\n\
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7460 done using the specified fill character (default is a space).");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7461
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7462 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7463 unicode_rjust(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7464 {
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
7465 Py_ssize_t width;
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7466 Py_UNICODE fillchar = ' ';
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7467
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
7468 if (!PyArg_ParseTuple(args, "n|O&:rjust", &width, convert_uc, &fillchar))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7469 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7470
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
7471 if (self->length >= width && PyUnicode_CheckExact(self)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7472 Py_INCREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7473 return (PyObject*) self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7474 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7475
31021
21a10c3600b5 Add optional fillchar argument to ljust(), rjust(), and center() string methods.
Raymond Hettinger <python@rcn.com>
parents: 30765
diff changeset
7476 return (PyObject*) pad(self, width - self->length, 0, fillchar);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7477 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7478
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7479 static PyObject*
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7480 unicode_slice(PyUnicodeObject *self, Py_ssize_t start, Py_ssize_t end)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7481 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7482 /* standard clamping */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7483 if (start < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7484 start = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7485 if (end < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7486 end = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7487 if (end > self->length)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7488 end = self->length;
19885
72e6eb653312 More on bug 460020: disable many optimizations of unicode subclasses.
Tim Peters <tim.peters@gmail.com>
parents: 19849
diff changeset
7489 if (start == 0 && end == self->length && PyUnicode_CheckExact(self)) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7490 /* full slice, return original string */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7491 Py_INCREF(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7492 return (PyObject*) self;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7493 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7494 if (start > end)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7495 start = end;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7496 /* copy slice */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7497 return (PyObject*) PyUnicode_FromUnicode(self->str + start,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7498 end - start);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7499 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7500
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7501 PyObject *PyUnicode_Split(PyObject *s,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7502 PyObject *sep,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7503 Py_ssize_t maxsplit)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7504 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7505 PyObject *result;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
7506
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7507 s = PyUnicode_FromObject(s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7508 if (s == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7509 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7510 if (sep != NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7511 sep = PyUnicode_FromObject(sep);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7512 if (sep == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7513 Py_DECREF(s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7514 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7515 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7516 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7517
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7518 result = split((PyUnicodeObject *)s, (PyUnicodeObject *)sep, maxsplit);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7519
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7520 Py_DECREF(s);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7521 Py_XDECREF(sep);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7522 return result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7523 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7524
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7525 PyDoc_STRVAR(split__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7526 "S.split([sep [,maxsplit]]) -> list of strings\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7527 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7528 Return a list of the words in S, using sep as the\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7529 delimiter string. If maxsplit is given, at most maxsplit\n\
46871
877783333eae #2809: elaborate str.split docstring a bit.
Georg Brandl <georg@python.org>
parents: 46391
diff changeset
7530 splits are done. If sep is not specified or is None, any\n\
46908
ed2c1888e143 #2809 followup: even better split docstring.
Georg Brandl <georg@python.org>
parents: 46871
diff changeset
7531 whitespace string is a separator and empty strings are\n\
ed2c1888e143 #2809 followup: even better split docstring.
Georg Brandl <georg@python.org>
parents: 46871
diff changeset
7532 removed from the result.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7533
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7534 static PyObject*
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7535 unicode_split(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7536 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7537 PyObject *substring = Py_None;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7538 Py_ssize_t maxcount = -1;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7539
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7540 if (!PyArg_ParseTuple(args, "|On:split", &substring, &maxcount))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7541 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7542
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7543 if (substring == Py_None)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7544 return split(self, NULL, maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7545 else if (PyUnicode_Check(substring))
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7546 return split(self, (PyUnicodeObject *)substring, maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7547 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7548 return PyUnicode_Split((PyObject *)self, substring, maxcount);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7549 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7550
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7551 PyObject *
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7552 PyUnicode_Partition(PyObject *str_in, PyObject *sep_in)
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7553 {
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7554 PyObject* str_obj;
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7555 PyObject* sep_obj;
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7556 PyObject* out;
38383
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7557
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7558 str_obj = PyUnicode_FromObject(str_in);
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7559 if (!str_obj)
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7560 return NULL;
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7561 sep_obj = PyUnicode_FromObject(sep_in);
38383
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7562 if (!sep_obj) {
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7563 Py_DECREF(str_obj);
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7564 return NULL;
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7565 }
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7566
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
7567 out = stringlib_partition(
38383
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7568 str_obj, PyUnicode_AS_UNICODE(str_obj), PyUnicode_GET_SIZE(str_obj),
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7569 sep_obj, PyUnicode_AS_UNICODE(sep_obj), PyUnicode_GET_SIZE(sep_obj)
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7570 );
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7571
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7572 Py_DECREF(sep_obj);
05744917b7c3 needforspeed: stringlib refactoring (in progress)
Fredrik Lundh <fredrik@pythonware.com>
parents: 38381
diff changeset
7573 Py_DECREF(str_obj);
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7574
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7575 return out;
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7576 }
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7577
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7578
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7579 PyObject *
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7580 PyUnicode_RPartition(PyObject *str_in, PyObject *sep_in)
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7581 {
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7582 PyObject* str_obj;
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7583 PyObject* sep_obj;
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7584 PyObject* out;
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7585
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7586 str_obj = PyUnicode_FromObject(str_in);
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7587 if (!str_obj)
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7588 return NULL;
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7589 sep_obj = PyUnicode_FromObject(sep_in);
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7590 if (!sep_obj) {
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7591 Py_DECREF(str_obj);
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7592 return NULL;
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7593 }
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7594
38400
33c328896f52 needforspeed: stringlib refactoring, continued. added count and
Fredrik Lundh <fredrik@pythonware.com>
parents: 38391
diff changeset
7595 out = stringlib_rpartition(
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7596 str_obj, PyUnicode_AS_UNICODE(str_obj), PyUnicode_GET_SIZE(str_obj),
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7597 sep_obj, PyUnicode_AS_UNICODE(sep_obj), PyUnicode_GET_SIZE(sep_obj)
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7598 );
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7599
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7600 Py_DECREF(sep_obj);
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7601 Py_DECREF(str_obj);
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7602
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7603 return out;
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7604 }
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7605
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7606 PyDoc_STRVAR(partition__doc__,
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7607 "S.partition(sep) -> (head, sep, tail)\n\
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7608 \n\
49431
2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49386
diff changeset
7609 Search for the separator sep in S, and return the part before it,\n\
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7610 the separator itself, and the part after it. If the separator is not\n\
49431
2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49386
diff changeset
7611 found, return S and two empty strings.");
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7612
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7613 static PyObject*
38338
958f91684209 needforspeed: use METH_O for argument handling, which made partition some
Fredrik Lundh <fredrik@pythonware.com>
parents: 38335
diff changeset
7614 unicode_partition(PyUnicodeObject *self, PyObject *separator)
958f91684209 needforspeed: use METH_O for argument handling, which made partition some
Fredrik Lundh <fredrik@pythonware.com>
parents: 38335
diff changeset
7615 {
38335
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7616 return PyUnicode_Partition((PyObject *)self, separator);
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7617 }
f9332ad4287a needforspeed: partition implementation, part two.
Fredrik Lundh <fredrik@pythonware.com>
parents: 38304
diff changeset
7618
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7619 PyDoc_STRVAR(rpartition__doc__,
39734
cfb77fff0240 Fix endcase for str.rpartition()
Raymond Hettinger <python@rcn.com>
parents: 39621
diff changeset
7620 "S.rpartition(sep) -> (tail, sep, head)\n\
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7621 \n\
49431
2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49386
diff changeset
7622 Search for the separator sep in S, starting at the end of S, and return\n\
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7623 the part before it, the separator itself, and the part after it. If the\n\
49431
2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49386
diff changeset
7624 separator is not found, return two empty strings and S.");
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7625
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7626 static PyObject*
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7627 unicode_rpartition(PyUnicodeObject *self, PyObject *separator)
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7628 {
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7629 return PyUnicode_RPartition((PyObject *)self, separator);
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7630 }
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7631
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7632 PyObject *PyUnicode_RSplit(PyObject *s,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7633 PyObject *sep,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7634 Py_ssize_t maxsplit)
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7635 {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7636 PyObject *result;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7637
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7638 s = PyUnicode_FromObject(s);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7639 if (s == NULL)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7640 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7641 if (sep != NULL) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7642 sep = PyUnicode_FromObject(sep);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7643 if (sep == NULL) {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7644 Py_DECREF(s);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7645 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7646 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7647 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7648
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7649 result = rsplit((PyUnicodeObject *)s, (PyUnicodeObject *)sep, maxsplit);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7650
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7651 Py_DECREF(s);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7652 Py_XDECREF(sep);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7653 return result;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7654 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7655
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7656 PyDoc_STRVAR(rsplit__doc__,
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7657 "S.rsplit([sep [,maxsplit]]) -> list of strings\n\
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7658 \n\
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7659 Return a list of the words in S, using sep as the\n\
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7660 delimiter string, starting at the end of the string and\n\
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7661 working to the front. If maxsplit is given, at most maxsplit\n\
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7662 splits are done. If sep is not specified, any whitespace string\n\
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7663 is a separator.");
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7664
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7665 static PyObject*
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7666 unicode_rsplit(PyUnicodeObject *self, PyObject *args)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7667 {
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7668 PyObject *substring = Py_None;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7669 Py_ssize_t maxcount = -1;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7670
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7671 if (!PyArg_ParseTuple(args, "|On:rsplit", &substring, &maxcount))
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7672 return NULL;
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7673
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7674 if (substring == Py_None)
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7675 return rsplit(self, NULL, maxcount);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7676 else if (PyUnicode_Check(substring))
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7677 return rsplit(self, (PyUnicodeObject *)substring, maxcount);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7678 else
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7679 return PyUnicode_RSplit((PyObject *)self, substring, maxcount);
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7680 }
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7681
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7682 PyDoc_STRVAR(splitlines__doc__,
50004
3c7eef187bea Merged revisions 67398,67423-67424,67432,67440-67441,67444-67445,67454,67457,67463 via svnmerge from
Georg Brandl <georg@python.org>
parents: 49431
diff changeset
7683 "S.splitlines([keepends]) -> list of strings\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7684 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7685 Return a list of the lines in S, breaking at line boundaries.\n\
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
7686 Line breaks are not included in the resulting list unless keepends\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7687 is given and true.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7688
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7689 static PyObject*
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7690 unicode_splitlines(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7691 {
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
7692 int keepends = 0;
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
7693
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
7694 if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7695 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7696
12369
68e48bb94071 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12329
diff changeset
7697 return PyUnicode_Splitlines((PyObject *)self, keepends);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7698 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7699
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7700 static
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7701 PyObject *unicode_str(PyUnicodeObject *self)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7702 {
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
7703 return PyUnicode_AsEncodedString((PyObject *)self, NULL, NULL);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7704 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7705
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7706 PyDoc_STRVAR(swapcase__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7707 "S.swapcase() -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7708 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7709 Return a copy of S with uppercase characters converted to lowercase\n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7710 and vice versa.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7711
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7712 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7713 unicode_swapcase(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7714 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7715 return fixup(self, fixswapcase);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7716 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7717
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7718 PyDoc_STRVAR(translate__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7719 "S.translate(table) -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7720 \n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7721 Return a copy of the string S, where all characters have been mapped\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7722 through the given translation table, which must be a mapping of\n\
25120
953ebe0c0c25 Change the unicode.translate docstring to document that
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
7723 Unicode ordinals to Unicode ordinals, Unicode strings or None.\n\
953ebe0c0c25 Change the unicode.translate docstring to document that
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
7724 Unmapped characters are left untouched. Characters mapped to None\n\
953ebe0c0c25 Change the unicode.translate docstring to document that
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
7725 are deleted.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7726
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7727 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7728 unicode_translate(PyUnicodeObject *self, PyObject *table)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7729 {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
7730 return PyUnicode_TranslateCharmap(self->str,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7731 self->length,
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
7732 table,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7733 "ignore");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7734 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7735
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7736 PyDoc_STRVAR(upper__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7737 "S.upper() -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7738 \n\
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7739 Return a copy of S converted to uppercase.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7740
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7741 static PyObject*
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7742 unicode_upper(PyUnicodeObject *self)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7743 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7744 return fixup(self, fixupper);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7745 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7746
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7747 PyDoc_STRVAR(zfill__doc__,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7748 "S.zfill(width) -> unicode\n\
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7749 \n\
49115
6f08278bf8b1 Fix varname in docstring. #3822.
Georg Brandl <georg@python.org>
parents: 48117
diff changeset
7750 Pad a numeric string S with zeros on the left, to fill a field\n\
6f08278bf8b1 Fix varname in docstring. #3822.
Georg Brandl <georg@python.org>
parents: 48117
diff changeset
7751 of the specified width. The string S is never truncated.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7752
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7753 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7754 unicode_zfill(PyUnicodeObject *self, PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7755 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7756 Py_ssize_t fill;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7757 PyUnicodeObject *u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7758
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7759 Py_ssize_t width;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7760 if (!PyArg_ParseTuple(args, "n:zfill", &width))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7761 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7762
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7763 if (self->length >= width) {
23051
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7764 if (PyUnicode_CheckExact(self)) {
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7765 Py_INCREF(self);
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7766 return (PyObject*) self;
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7767 }
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7768 else
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7769 return PyUnicode_FromUnicode(
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7770 PyUnicode_AS_UNICODE(self),
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7771 PyUnicode_GET_SIZE(self)
19ea00feda33 Return the orginal string only if it's a real str or unicode
Walter Dörwald <walter@livinglogic.de>
parents: 23045
diff changeset
7772 );
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7773 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7774
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7775 fill = width - self->length;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7776
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7777 u = pad(self, fill, 0, '0');
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7778
23045
8b1268bc371a Apply the second version of SF patch http://www.python.org/sf/536241
Walter Dörwald <walter@livinglogic.de>
parents: 22981
diff changeset
7779 if (u == NULL)
8b1268bc371a Apply the second version of SF patch http://www.python.org/sf/536241
Walter Dörwald <walter@livinglogic.de>
parents: 22981
diff changeset
7780 return NULL;
8b1268bc371a Apply the second version of SF patch http://www.python.org/sf/536241
Walter Dörwald <walter@livinglogic.de>
parents: 22981
diff changeset
7781
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7782 if (u->str[fill] == '+' || u->str[fill] == '-') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7783 /* move sign to beginning of string */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7784 u->str[0] = u->str[fill];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7785 u->str[fill] = '0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7786 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7787
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7788 return (PyObject*) u;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7789 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7790
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7791 #if 0
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7792 static PyObject*
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
7793 free_listsize(PyUnicodeObject *self)
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
7794 {
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
7795 return PyInt_FromLong(numfree);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7796 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7797 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7798
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7799 PyDoc_STRVAR(startswith__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7800 "S.startswith(prefix[, start[, end]]) -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7801 \n\
28559
aff080f2efb4 Reformat a few docstrings that caused line wraps in help() output.
Guido van Rossum <guido@python.org>
parents: 28494
diff changeset
7802 Return True if S starts with the specified prefix, False otherwise.\n\
aff080f2efb4 Reformat a few docstrings that caused line wraps in help() output.
Guido van Rossum <guido@python.org>
parents: 28494
diff changeset
7803 With optional start, test S beginning at that position.\n\
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7804 With optional end, stop comparing S at that position.\n\
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7805 prefix can also be a tuple of strings to try.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7806
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7807 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7808 unicode_startswith(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7809 PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7810 {
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7811 PyObject *subobj;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7812 PyUnicodeObject *substring;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7813 Py_ssize_t start = 0;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
7814 Py_ssize_t end = PY_SSIZE_T_MAX;
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7815 int result;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7816
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7817 if (!PyArg_ParseTuple(args, "O|O&O&:startswith", &subobj,
12629
3efeaa7521e4 Trent Mick:
Guido van Rossum <guido@python.org>
parents: 12582
diff changeset
7818 _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7819 return NULL;
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7820 if (PyTuple_Check(subobj)) {
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7821 Py_ssize_t i;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7822 for (i = 0; i < PyTuple_GET_SIZE(subobj); i++) {
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7823 substring = (PyUnicodeObject *)PyUnicode_FromObject(
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7824 PyTuple_GET_ITEM(subobj, i));
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7825 if (substring == NULL)
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7826 return NULL;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7827 result = tailmatch(self, substring, start, end, -1);
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7828 Py_DECREF(substring);
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7829 if (result) {
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7830 Py_RETURN_TRUE;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7831 }
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7832 }
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7833 /* nothing matched */
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7834 Py_RETURN_FALSE;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7835 }
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7836 substring = (PyUnicodeObject *)PyUnicode_FromObject(subobj);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7837 if (substring == NULL)
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7838 return NULL;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7839 result = tailmatch(self, substring, start, end, -1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7840 Py_DECREF(substring);
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7841 return PyBool_FromLong(result);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7842 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7843
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7844
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
7845 PyDoc_STRVAR(endswith__doc__,
22785
aa8d59823add Add the 'bool' type and its values 'False' and 'True', as described in
Guido van Rossum <guido@python.org>
parents: 22521
diff changeset
7846 "S.endswith(suffix[, start[, end]]) -> bool\n\
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7847 \n\
28559
aff080f2efb4 Reformat a few docstrings that caused line wraps in help() output.
Guido van Rossum <guido@python.org>
parents: 28494
diff changeset
7848 Return True if S ends with the specified suffix, False otherwise.\n\
aff080f2efb4 Reformat a few docstrings that caused line wraps in help() output.
Guido van Rossum <guido@python.org>
parents: 28494
diff changeset
7849 With optional start, test S beginning at that position.\n\
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7850 With optional end, stop comparing S at that position.\n\
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7851 suffix can also be a tuple of strings to try.");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7852
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7853 static PyObject *
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7854 unicode_endswith(PyUnicodeObject *self,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7855 PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7856 {
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7857 PyObject *subobj;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7858 PyUnicodeObject *substring;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
7859 Py_ssize_t start = 0;
37662
0678ce9fb6b2 Change more ints to Py_ssize_t.
Martin v. Löwis <martin@v.loewis.de>
parents: 37661
diff changeset
7860 Py_ssize_t end = PY_SSIZE_T_MAX;
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7861 int result;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7862
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7863 if (!PyArg_ParseTuple(args, "O|O&O&:endswith", &subobj,
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7864 _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7865 return NULL;
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7866 if (PyTuple_Check(subobj)) {
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7867 Py_ssize_t i;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7868 for (i = 0; i < PyTuple_GET_SIZE(subobj); i++) {
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7869 substring = (PyUnicodeObject *)PyUnicode_FromObject(
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7870 PyTuple_GET_ITEM(subobj, i));
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7871 if (substring == NULL)
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7872 return NULL;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7873 result = tailmatch(self, substring, start, end, +1);
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7874 Py_DECREF(substring);
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7875 if (result) {
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7876 Py_RETURN_TRUE;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7877 }
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7878 }
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7879 Py_RETURN_FALSE;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7880 }
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7881 substring = (PyUnicodeObject *)PyUnicode_FromObject(subobj);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7882 if (substring == NULL)
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7883 return NULL;
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7884
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7885 result = tailmatch(self, substring, start, end, +1);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7886 Py_DECREF(substring);
38697
1cafbcd7f55c RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument.
Georg Brandl <georg@python.org>
parents: 38587
diff changeset
7887 return PyBool_FromLong(result);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7888 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7889
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7890
45381
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7891 /* Implements do_string_format, which is unicode because of stringlib */
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7892 #include "stringlib/string_format.h"
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7893
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7894 PyDoc_STRVAR(format__doc__,
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7895 "S.format(*args, **kwargs) -> unicode\n\
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7896 \n\
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7897 ");
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7898
47392
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7899 static PyObject *
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7900 unicode__format__(PyObject *self, PyObject *args)
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7901 {
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7902 PyObject *format_spec;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7903 PyObject *result = NULL;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7904 PyObject *tmp = NULL;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7905
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7906 /* If 2.x, convert format_spec to the same type as value */
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7907 /* This is to allow things like u''.format('') */
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7908 if (!PyArg_ParseTuple(args, "O:__format__", &format_spec))
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7909 goto done;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7910 if (!(PyBytes_Check(format_spec) || PyUnicode_Check(format_spec))) {
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7911 PyErr_Format(PyExc_TypeError, "__format__ arg must be str "
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7912 "or unicode, not %s", Py_TYPE(format_spec)->tp_name);
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7913 goto done;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7914 }
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7915 tmp = PyObject_Unicode(format_spec);
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7916 if (tmp == NULL)
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7917 goto done;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7918 format_spec = tmp;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7919
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7920 result = _PyUnicode_FormatAdvanced(self,
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7921 PyUnicode_AS_UNICODE(format_spec),
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7922 PyUnicode_GET_SIZE(format_spec));
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7923 done:
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7924 Py_XDECREF(tmp);
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7925 return result;
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7926 }
47ef9a806ce9 Refactor and clean up str.format() code (and helpers) in advance of optimizations.
Eric Smith <eric@trueblade.com>
parents: 47388
diff changeset
7927
45381
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7928 PyDoc_STRVAR(p_format__doc__,
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7929 "S.__format__(format_spec) -> unicode\n\
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7930 \n\
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7931 ");
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7932
47543
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7933 static PyObject *
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7934 unicode__sizeof__(PyUnicodeObject *v)
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7935 {
47922
7cfc28ac0715 Added additional __sizeof__ implementations and addressed comments made in
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47543
diff changeset
7936 return PyInt_FromSsize_t(sizeof(PyUnicodeObject) +
7cfc28ac0715 Added additional __sizeof__ implementations and addressed comments made in
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47543
diff changeset
7937 sizeof(Py_UNICODE) * (v->length + 1));
47543
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7938 }
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7939
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7940 PyDoc_STRVAR(sizeof__doc__,
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7941 "S.__sizeof__() -> size of S in memory, in bytes\n\
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7942 \n\
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
7943 ");
27521
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7944
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7945 static PyObject *
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7946 unicode_getnewargs(PyUnicodeObject *v)
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7947 {
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7948 return Py_BuildValue("(u#)", v->str, v->length);
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7949 }
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7950
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
7951
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7952 static PyMethodDef unicode_methods[] = {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7953
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7954 /* Order is according to common usage: often used methods should
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7955 appear first, since lookup is done sequentially. */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7956
37306
ce56c5e3cb20 That one was a mistake.
Georg Brandl <georg@python.org>
parents: 37304
diff changeset
7957 {"encode", (PyCFunction) unicode_encode, METH_VARARGS, encode__doc__},
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7958 {"replace", (PyCFunction) unicode_replace, METH_VARARGS, replace__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7959 {"split", (PyCFunction) unicode_split, METH_VARARGS, split__doc__},
31199
504943c89193 Add rsplit method for str and unicode builtin types.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31042
diff changeset
7960 {"rsplit", (PyCFunction) unicode_rsplit, METH_VARARGS, rsplit__doc__},
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7961 {"join", (PyCFunction) unicode_join, METH_O, join__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7962 {"capitalize", (PyCFunction) unicode_capitalize, METH_NOARGS, capitalize__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7963 {"title", (PyCFunction) unicode_title, METH_NOARGS, title__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7964 {"center", (PyCFunction) unicode_center, METH_VARARGS, center__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7965 {"count", (PyCFunction) unicode_count, METH_VARARGS, count__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7966 {"expandtabs", (PyCFunction) unicode_expandtabs, METH_VARARGS, expandtabs__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7967 {"find", (PyCFunction) unicode_find, METH_VARARGS, find__doc__},
38338
958f91684209 needforspeed: use METH_O for argument handling, which made partition some
Fredrik Lundh <fredrik@pythonware.com>
parents: 38335
diff changeset
7968 {"partition", (PyCFunction) unicode_partition, METH_O, partition__doc__},
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7969 {"index", (PyCFunction) unicode_index, METH_VARARGS, index__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7970 {"ljust", (PyCFunction) unicode_ljust, METH_VARARGS, ljust__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7971 {"lower", (PyCFunction) unicode_lower, METH_NOARGS, lower__doc__},
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7972 {"lstrip", (PyCFunction) unicode_lstrip, METH_VARARGS, lstrip__doc__},
32551
f46d49e2e0f0 Allow string and unicode return types from .encode()/.decode()
Marc-André Lemburg <mal@egenix.com>
parents: 32363
diff changeset
7973 {"decode", (PyCFunction) unicode_decode, METH_VARARGS, decode__doc__},
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7974 /* {"maketrans", (PyCFunction) unicode_maketrans, METH_VARARGS, maketrans__doc__}, */
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7975 {"rfind", (PyCFunction) unicode_rfind, METH_VARARGS, rfind__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7976 {"rindex", (PyCFunction) unicode_rindex, METH_VARARGS, rindex__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7977 {"rjust", (PyCFunction) unicode_rjust, METH_VARARGS, rjust__doc__},
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7978 {"rstrip", (PyCFunction) unicode_rstrip, METH_VARARGS, rstrip__doc__},
38390
76807902a2f5 needforspeed: added rpartition implementation
Fredrik Lundh <fredrik@pythonware.com>
parents: 38383
diff changeset
7979 {"rpartition", (PyCFunction) unicode_rpartition, METH_O, rpartition__doc__},
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7980 {"splitlines", (PyCFunction) unicode_splitlines, METH_VARARGS, splitlines__doc__},
23225
e639e34ef901 Apply patch diff.txt from SF feature request
Walter Dörwald <walter@livinglogic.de>
parents: 23205
diff changeset
7981 {"strip", (PyCFunction) unicode_strip, METH_VARARGS, strip__doc__},
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7982 {"swapcase", (PyCFunction) unicode_swapcase, METH_NOARGS, swapcase__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7983 {"translate", (PyCFunction) unicode_translate, METH_O, translate__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7984 {"upper", (PyCFunction) unicode_upper, METH_NOARGS, upper__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7985 {"startswith", (PyCFunction) unicode_startswith, METH_VARARGS, startswith__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7986 {"endswith", (PyCFunction) unicode_endswith, METH_VARARGS, endswith__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7987 {"islower", (PyCFunction) unicode_islower, METH_NOARGS, islower__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7988 {"isupper", (PyCFunction) unicode_isupper, METH_NOARGS, isupper__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7989 {"istitle", (PyCFunction) unicode_istitle, METH_NOARGS, istitle__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7990 {"isspace", (PyCFunction) unicode_isspace, METH_NOARGS, isspace__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7991 {"isdecimal", (PyCFunction) unicode_isdecimal, METH_NOARGS, isdecimal__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7992 {"isdigit", (PyCFunction) unicode_isdigit, METH_NOARGS, isdigit__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7993 {"isnumeric", (PyCFunction) unicode_isnumeric, METH_NOARGS, isnumeric__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7994 {"isalpha", (PyCFunction) unicode_isalpha, METH_NOARGS, isalpha__doc__},
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
7995 {"isalnum", (PyCFunction) unicode_isalnum, METH_NOARGS, isalnum__doc__},
23045
8b1268bc371a Apply the second version of SF patch http://www.python.org/sf/536241
Walter Dörwald <walter@livinglogic.de>
parents: 22981
diff changeset
7996 {"zfill", (PyCFunction) unicode_zfill, METH_VARARGS, zfill__doc__},
45381
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7997 {"format", (PyCFunction) do_string_format, METH_VARARGS | METH_KEYWORDS, format__doc__},
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7998 {"__format__", (PyCFunction) unicode__format__, METH_VARARGS, p_format__doc__},
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
7999 {"_formatter_field_name_split", (PyCFunction) formatter_field_name_split, METH_NOARGS},
b7231bf6bfb7 Backport of PEP 3101, Advanced String Formatting, from py3k.
Eric Smith <eric@trueblade.com>
parents: 45341
diff changeset
8000 {"_formatter_parser", (PyCFunction) formatter_parser, METH_NOARGS},
47543
296ab1038b7f Issue 3048: Fixed sys.getsizeof for unicode objects.
Robert Schuppenies <okkotonushi@googlemail.com>
parents: 47527
diff changeset
8001 {"__sizeof__", (PyCFunction) unicode__sizeof__, METH_NOARGS, sizeof__doc__},
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8002 #if 0
19301
b9701538f827 Patch #427190: Implement and use METH_NOARGS and METH_O.
Martin v. Löwis <martin@v.loewis.de>
parents: 19121
diff changeset
8003 {"capwords", (PyCFunction) unicode_capwords, METH_NOARGS, capwords__doc__},
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8004 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8005
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8006 #if 0
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8007 /* This one is just used for debugging the implementation. */
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
8008 {"freelistsize", (PyCFunction) free_listsize, METH_NOARGS},
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8009 #endif
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8010
27521
17bfd5dbc0f1 Implement appropriate __getnewargs__ for all immutable subclassable builtin
Guido van Rossum <guido@python.org>
parents: 27175
diff changeset
8011 {"__getnewargs__", (PyCFunction)unicode_getnewargs, METH_NOARGS},
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8012 {NULL, NULL}
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8013 };
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8014
26274
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8015 static PyObject *
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8016 unicode_mod(PyObject *v, PyObject *w)
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8017 {
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8018 if (!PyUnicode_Check(v)) {
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8019 Py_INCREF(Py_NotImplemented);
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8020 return Py_NotImplemented;
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8021 }
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8022 return PyUnicode_Format(v, w);
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8023 }
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8024
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8025 static PyNumberMethods unicode_as_number = {
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8026 0, /*nb_add*/
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8027 0, /*nb_subtract*/
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8028 0, /*nb_multiply*/
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8029 0, /*nb_divide*/
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8030 unicode_mod, /*nb_remainder*/
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8031 };
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8032
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8033 static PySequenceMethods unicode_as_sequence = {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8034 (lenfunc) unicode_length, /* sq_length */
37304
79a0f080f898 Remove unnecessary casts in type object initializers.
Georg Brandl <georg@python.org>
parents: 36945
diff changeset
8035 PyUnicode_Concat, /* sq_concat */
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8036 (ssizeargfunc) unicode_repeat, /* sq_repeat */
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8037 (ssizeargfunc) unicode_getitem, /* sq_item */
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8038 (ssizessizeargfunc) unicode_slice, /* sq_slice */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8039 0, /* sq_ass_item */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8040 0, /* sq_ass_slice */
37304
79a0f080f898 Remove unnecessary casts in type object initializers.
Georg Brandl <georg@python.org>
parents: 36945
diff changeset
8041 PyUnicode_Contains, /* sq_contains */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8042 };
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8043
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8044 static PyObject*
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8045 unicode_subscript(PyUnicodeObject* self, PyObject* item)
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8046 {
39491
2312e178b34d Correct an accidentally removed previous patch.
Marc-André Lemburg <mal@egenix.com>
parents: 39487
diff changeset
8047 if (PyIndex_Check(item)) {
2312e178b34d Correct an accidentally removed previous patch.
Marc-André Lemburg <mal@egenix.com>
parents: 39487
diff changeset
8048 Py_ssize_t i = PyNumber_AsSsize_t(item, PyExc_IndexError);
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8049 if (i == -1 && PyErr_Occurred())
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8050 return NULL;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8051 if (i < 0)
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
8052 i += PyUnicode_GET_SIZE(self);
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8053 return unicode_getitem(self, i);
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8054 } else if (PySlice_Check(item)) {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8055 Py_ssize_t start, stop, step, slicelength, cur, i;
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8056 Py_UNICODE* source_buf;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8057 Py_UNICODE* result_buf;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8058 PyObject* result;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8059
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
8060 if (PySlice_GetIndicesEx((PySliceObject*)item, PyUnicode_GET_SIZE(self),
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8061 &start, &stop, &step, &slicelength) < 0) {
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8062 return NULL;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8063 }
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8064
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8065 if (slicelength <= 0) {
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8066 return PyUnicode_FromUnicode(NULL, 0);
42951
9620134010a5 Improve extended slicing support in builtin types and classes. Specifically:
Thomas Wouters <thomas@python.org>
parents: 42652
diff changeset
8067 } else if (start == 0 && step == 1 && slicelength == self->length &&
9620134010a5 Improve extended slicing support in builtin types and classes. Specifically:
Thomas Wouters <thomas@python.org>
parents: 42652
diff changeset
8068 PyUnicode_CheckExact(self)) {
9620134010a5 Improve extended slicing support in builtin types and classes. Specifically:
Thomas Wouters <thomas@python.org>
parents: 42652
diff changeset
8069 Py_INCREF(self);
9620134010a5 Improve extended slicing support in builtin types and classes. Specifically:
Thomas Wouters <thomas@python.org>
parents: 42652
diff changeset
8070 return (PyObject *)self;
9620134010a5 Improve extended slicing support in builtin types and classes. Specifically:
Thomas Wouters <thomas@python.org>
parents: 42652
diff changeset
8071 } else if (step == 1) {
9620134010a5 Improve extended slicing support in builtin types and classes. Specifically:
Thomas Wouters <thomas@python.org>
parents: 42652
diff changeset
8072 return PyUnicode_FromUnicode(self->str + start, slicelength);
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8073 } else {
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8074 source_buf = PyUnicode_AS_UNICODE((PyObject*)self);
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
8075 result_buf = (Py_UNICODE *)PyObject_MALLOC(slicelength*
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
8076 sizeof(Py_UNICODE));
36203
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
8077
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
8078 if (result_buf == NULL)
44cfca6136c6 Stop maintaining the buildno file.
Martin v. Löwis <martin@v.loewis.de>
parents: 36056
diff changeset
8079 return PyErr_NoMemory();
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8080
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8081 for (cur = start, i = 0; i < slicelength; cur += step, i++) {
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8082 result_buf[i] = source_buf[cur];
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8083 }
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8084
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8085 result = PyUnicode_FromUnicode(result_buf, slicelength);
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
8086 PyObject_FREE(result_buf);
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8087 return result;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8088 }
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8089 } else {
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8090 PyErr_SetString(PyExc_TypeError, "string indices must be integers");
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8091 return NULL;
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8092 }
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8093 }
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8094
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8095 static PyMappingMethods unicode_as_mapping = {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8096 (lenfunc)unicode_length, /* mp_length */
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8097 (binaryfunc)unicode_subscript, /* mp_subscript */
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8098 (objobjargproc)0, /* mp_ass_subscript */
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8099 };
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8100
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8101 static Py_ssize_t
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8102 unicode_buffer_getreadbuf(PyUnicodeObject *self,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8103 Py_ssize_t index,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8104 const void **ptr)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8105 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8106 if (index != 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8107 PyErr_SetString(PyExc_SystemError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8108 "accessing non-existent unicode segment");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8109 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8110 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8111 *ptr = (void *) self->str;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8112 return PyUnicode_GET_DATA_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8113 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8114
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8115 static Py_ssize_t
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8116 unicode_buffer_getwritebuf(PyUnicodeObject *self, Py_ssize_t index,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8117 const void **ptr)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8118 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8119 PyErr_SetString(PyExc_TypeError,
23837
359a57e96d5e Fix typo in exception message
Neal Norwitz <nnorwitz@gmail.com>
parents: 23832
diff changeset
8120 "cannot use unicode as modifiable buffer");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8121 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8122 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8123
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8124 static int
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8125 unicode_buffer_getsegcount(PyUnicodeObject *self,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8126 Py_ssize_t *lenp)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8127 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8128 if (lenp)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8129 *lenp = PyUnicode_GET_DATA_SIZE(self);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8130 return 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8131 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8132
36519
d6d415df0df0 Use Py_ssize_t for counts and sizes.
Martin v. Löwis <martin@v.loewis.de>
parents: 36510
diff changeset
8133 static Py_ssize_t
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8134 unicode_buffer_getcharbuf(PyUnicodeObject *self,
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8135 Py_ssize_t index,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8136 const void **ptr)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8137 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8138 PyObject *str;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8139
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8140 if (index != 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8141 PyErr_SetString(PyExc_SystemError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8142 "accessing non-existent unicode segment");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8143 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8144 }
13837
2badb322355f This patch finalizes the move from UTF-8 to a default encoding in
Marc-André Lemburg <mal@egenix.com>
parents: 13828
diff changeset
8145 str = _PyUnicode_AsDefaultEncodedString((PyObject *)self, NULL);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8146 if (str == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8147 return -1;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8148 *ptr = (void *) PyString_AS_STRING(str);
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8149 return PyString_GET_SIZE(str);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8150 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8151
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8152 /* Helpers for PyUnicode_Format() */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8153
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8154 static PyObject *
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8155 getnextarg(PyObject *args, Py_ssize_t arglen, Py_ssize_t *p_argidx)
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8156 {
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8157 Py_ssize_t argidx = *p_argidx;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8158 if (argidx < arglen) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8159 (*p_argidx)++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8160 if (arglen < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8161 return args;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8162 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8163 return PyTuple_GetItem(args, argidx);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8164 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8165 PyErr_SetString(PyExc_TypeError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8166 "not enough arguments for format string");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8167 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8168 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8169
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8170 #define F_LJUST (1<<0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8171 #define F_SIGN (1<<1)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8172 #define F_BLANK (1<<2)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8173 #define F_ALT (1<<3)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8174 #define F_ZERO (1<<4)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8175
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8176 static Py_ssize_t
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8177 strtounicode(Py_UNICODE *buffer, const char *charbuffer)
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8178 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8179 register Py_ssize_t i;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8180 Py_ssize_t len = strlen(charbuffer);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8181 for (i = len - 1; i >= 0; i--)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8182 buffer[i] = (Py_UNICODE) charbuffer[i];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8183
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8184 return len;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8185 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8186
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8187 static int
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8188 doubletounicode(Py_UNICODE *buffer, size_t len, const char *format, double x)
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8189 {
36508
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8190 Py_ssize_t result;
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8191
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8192 PyOS_ascii_formatd((char *)buffer, len, format, x);
36508
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8193 result = strtounicode(buffer, (char *)buffer);
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8194 return Py_SAFE_DOWNCAST(result, Py_ssize_t, int);
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8195 }
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8196
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8197 static int
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8198 longtounicode(Py_UNICODE *buffer, size_t len, const char *format, long x)
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8199 {
36508
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8200 Py_ssize_t result;
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8201
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8202 PyOS_snprintf((char *)buffer, len, format, x);
36508
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8203 result = strtounicode(buffer, (char *)buffer);
7322dccba224 doubletounicode(), longtounicode():
Tim Peters <tim.peters@gmail.com>
parents: 36506
diff changeset
8204 return Py_SAFE_DOWNCAST(result, Py_ssize_t, int);
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8205 }
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8206
24736
ff05eef3a6d1 Implement stage B0 of PEP 237: add warnings for operations that
Guido van Rossum <guido@python.org>
parents: 24703
diff changeset
8207 /* XXX To save some code duplication, formatfloat/long/int could have been
ff05eef3a6d1 Implement stage B0 of PEP 237: add warnings for operations that
Guido van Rossum <guido@python.org>
parents: 24703
diff changeset
8208 shared with stringobject.c, converting from 8-bit to Unicode after the
ff05eef3a6d1 Implement stage B0 of PEP 237: add warnings for operations that
Guido van Rossum <guido@python.org>
parents: 24703
diff changeset
8209 formatting is done. */
ff05eef3a6d1 Implement stage B0 of PEP 237: add warnings for operations that
Guido van Rossum <guido@python.org>
parents: 24703
diff changeset
8210
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8211 static int
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8212 formatfloat(Py_UNICODE *buf,
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8213 size_t buflen,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8214 int flags,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8215 int prec,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8216 int type,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8217 PyObject *v)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8218 {
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8219 /* fmt = '%#.' + `prec` + `type`
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8220 worst case length = 3 + 10 (len of INT_MAX) + 1 = 14 (use 20)*/
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8221 char fmt[20];
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8222 double x;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8223
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8224 x = PyFloat_AsDouble(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8225 if (x == -1.0 && PyErr_Occurred())
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8226 return -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8227 if (prec < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8228 prec = 6;
48003
5c33009994b4 Backed out r65069, pending fixing it in Windows.
Eric Smith <eric@trueblade.com>
parents: 48001
diff changeset
8229 if (type == 'f' && (fabs(x) / 1e25) >= 1e25)
5c33009994b4 Backed out r65069, pending fixing it in Windows.
Eric Smith <eric@trueblade.com>
parents: 48001
diff changeset
8230 type = 'g';
26825
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8231 /* Worst case length calc to ensure no buffer overrun:
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8232
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8233 'g' formats:
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8234 fmt = %#.<prec>g
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8235 buf = '-' + [0-9]*prec + '.' + 'e+' + (longest exp
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8236 for any double rep.)
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8237 len = 1 + prec + 1 + 2 + 5 = 9 + prec
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8238
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8239 'f' formats:
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8240 buf = '-' + [0-9]*x + '.' + [0-9]*prec (with x < 50)
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8241 len = 1 + 50 + 1 + prec = 52 + prec
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8242
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8243 If prec=0 the effective precision is 1 (the leading digit is
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8244 always given), therefore increase the length by one.
26825
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8245
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8246 */
42216
86d1dbfc5833 Patch #1673759: add a missing overflow check when formatting floats
Georg Brandl <georg@python.org>
parents: 42024
diff changeset
8247 if (((type == 'g' || type == 'G') &&
86d1dbfc5833 Patch #1673759: add a missing overflow check when formatting floats
Georg Brandl <georg@python.org>
parents: 42024
diff changeset
8248 buflen <= (size_t)10 + (size_t)prec) ||
48003
5c33009994b4 Backed out r65069, pending fixing it in Windows.
Eric Smith <eric@trueblade.com>
parents: 48001
diff changeset
8249 (type == 'f' && buflen <= (size_t)53 + (size_t)prec)) {
26825
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8250 PyErr_SetString(PyExc_OverflowError,
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8251 "formatted float is too long (precision too large?)");
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8252 return -1;
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8253 }
21082
c8d775b9ff92 formatfloat(), formatint(): Conversion of sprintf() to PyOS_snprintf()
Barry Warsaw <barry@python.org>
parents: 21067
diff changeset
8254 PyOS_snprintf(fmt, sizeof(fmt), "%%%s.%d%c",
26825
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8255 (flags&F_ALT) ? "#" : "",
e153961f961f Patch for bug #659709: bogus computation of float length
Marc-André Lemburg <mal@egenix.com>
parents: 26274
diff changeset
8256 prec, type);
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8257 return doubletounicode(buf, buflen, fmt, x);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8258 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8259
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8260 static PyObject*
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8261 formatlong(PyObject *val, int flags, int prec, int type)
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8262 {
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8263 char *buf;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8264 int i, len;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8265 PyObject *str; /* temporary string object. */
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8266 PyUnicodeObject *result;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8267
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8268 str = _PyString_FormatLong(val, flags, prec, type, &buf, &len);
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8269 if (!str)
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8270 return NULL;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8271 result = _PyUnicode_New(len);
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
8272 if (!result) {
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
8273 Py_DECREF(str);
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
8274 return NULL;
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
8275 }
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8276 for (i = 0; i < len; i++)
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8277 result->str[i] = buf[i];
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8278 result->str[len] = 0;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8279 Py_DECREF(str);
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8280 return (PyObject*)result;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8281 }
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8282
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8283 static int
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8284 formatint(Py_UNICODE *buf,
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8285 size_t buflen,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8286 int flags,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8287 int prec,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8288 int type,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8289 PyObject *v)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8290 {
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8291 /* fmt = '%#.' + `prec` + 'l' + `type`
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8292 * worst case length = 3 + 19 (worst len of INT_MAX on 64-bit machine)
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8293 * + 1 + 1
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8294 * = 24
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8295 */
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8296 char fmt[64]; /* plenty big enough! */
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8297 char *sign;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8298 long x;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8299
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8300 x = PyInt_AsLong(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8301 if (x == -1 && PyErr_Occurred())
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8302 return -1;
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8303 if (x < 0 && type == 'u') {
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8304 type = 'd';
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8305 }
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8306 if (x < 0 && (type == 'x' || type == 'X' || type == 'o'))
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8307 sign = "-";
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8308 else
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8309 sign = "";
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8310 if (prec < 0)
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8311 prec = 1;
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8312
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8313 /* buf = '+'/'-'/'' + '0'/'0x'/'' + '[0-9]'*max(prec, len(x in octal))
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8314 * worst case buf = '-0x' + [0-9]*prec, where prec >= 11
17450
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8315 */
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8316 if (buflen <= 14 || buflen <= (size_t)3 + (size_t)prec) {
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8317 PyErr_SetString(PyExc_OverflowError,
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8318 "formatted integer is too long (precision too large?)");
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8319 return -1;
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8320 }
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8321
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8322 if ((flags & F_ALT) &&
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8323 (type == 'x' || type == 'X')) {
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8324 /* When converting under %#x or %#X, there are a number
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8325 * of issues that cause pain:
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8326 * - when 0 is being converted, the C standard leaves off
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8327 * the '0x' or '0X', which is inconsistent with other
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8328 * %#x/%#X conversions and inconsistent with Python's
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8329 * hex() function
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8330 * - there are platforms that violate the standard and
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8331 * convert 0 with the '0x' or '0X'
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8332 * (Metrowerks, Compaq Tru64)
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8333 * - there are platforms that give '0x' when converting
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8334 * under %#X, but convert 0 in accordance with the
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8335 * standard (OS/2 EMX)
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8336 *
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8337 * We can achieve the desired consistency by inserting our
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8338 * own '0x' or '0X' prefix, and substituting %x/%X in place
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8339 * of %#x/%#X.
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8340 *
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8341 * Note that this is the same approach as used in
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8342 * formatint() in stringobject.c
22053
49197fd76362 OS/2 EMX port changes (Objects part of patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 21886
diff changeset
8343 */
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8344 PyOS_snprintf(fmt, sizeof(fmt), "%s0%c%%.%dl%c",
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8345 sign, type, prec, type);
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8346 }
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8347 else {
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8348 PyOS_snprintf(fmt, sizeof(fmt), "%s%%%s.%dl%c",
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8349 sign, (flags&F_ALT) ? "#" : "",
22087
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8350 prec, type);
5b24a6ae2dc4 %#x/%#X format conversion cleanup (see patch #450267):
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22053
diff changeset
8351 }
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8352 if (sign[0])
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8353 return longtounicode(buf, buflen, fmt, -x);
31042
9c846c3df6f3 - Removed FutureWarnings related to hex/oct literals and conversions
Guido van Rossum <guido@python.org>
parents: 31021
diff changeset
8354 else
36261
303dea847639 - Patch #1400181, fix unicode string formatting to not use the locale.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36236
diff changeset
8355 return longtounicode(buf, buflen, fmt, x);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8356 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8357
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8358 static int
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8359 formatchar(Py_UNICODE *buf,
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8360 size_t buflen,
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8361 PyObject *v)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8362 {
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8363 /* presume that the buffer is at least 2 characters long */
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8364 if (PyUnicode_Check(v)) {
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8365 if (PyUnicode_GET_SIZE(v) != 1)
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8366 goto onError;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8367 buf[0] = PyUnicode_AS_UNICODE(v)[0];
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8368 }
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8369
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8370 else if (PyString_Check(v)) {
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8371 if (PyString_GET_SIZE(v) != 1)
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8372 goto onError;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8373 buf[0] = (Py_UNICODE)PyString_AS_STRING(v)[0];
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8374 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8375
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8376 else {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8377 /* Integer input truncated to a character */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8378 long x;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8379 x = PyInt_AsLong(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8380 if (x == -1 && PyErr_Occurred())
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8381 goto onError;
24737
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8382 #ifdef Py_UNICODE_WIDE
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8383 if (x < 0 || x > 0x10ffff) {
28494
32bea41fef68 Change formatchar(), so that u"%c" % 0xffffffff now raises
Walter Dörwald <walter@livinglogic.de>
parents: 28242
diff changeset
8384 PyErr_SetString(PyExc_OverflowError,
24737
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8385 "%c arg not in range(0x110000) "
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8386 "(wide Python build)");
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8387 return -1;
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8388 }
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8389 #else
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8390 if (x < 0 || x > 0xffff) {
28494
32bea41fef68 Change formatchar(), so that u"%c" % 0xffffffff now raises
Walter Dörwald <walter@livinglogic.de>
parents: 28242
diff changeset
8391 PyErr_SetString(PyExc_OverflowError,
24737
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8392 "%c arg not in range(0x10000) "
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8393 "(narrow Python build)");
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8394 return -1;
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8395 }
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8396 #endif
eaa87632b1e8 Add C API PyUnicode_FromOrdinal() which exposes unichr() at C level.
Marc-André Lemburg <mal@egenix.com>
parents: 24736
diff changeset
8397 buf[0] = (Py_UNICODE) x;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8398 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8399 buf[1] = '\0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8400 return 1;
12899
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8401
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8402 onError:
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8403 PyErr_SetString(PyExc_TypeError,
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8404 "%c requires int or char");
9df91f91e5cc Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12876
diff changeset
8405 return -1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8406 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8407
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8408 /* fmt%(v1,v2,...) is roughly equivalent to sprintf(fmt, v1, v2, ...)
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8409
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8410 FORMATBUFLEN is the length of the buffer in which the floats, ints, &
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8411 chars are formatted. XXX This is a magic number. Each formatting
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8412 routine does bounds checking to ensure no overflow, but a better
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8413 solution may be to malloc a buffer of appropriate size for each
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8414 format. For now, the current solution is sufficient.
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8415 */
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8416 #define FORMATBUFLEN (size_t)120
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8417
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8418 PyObject *PyUnicode_Format(PyObject *format,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8419 PyObject *args)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8420 {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8421 Py_UNICODE *fmt, *res;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8422 Py_ssize_t fmtcnt, rescnt, reslen, arglen, argidx;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8423 int args_owned = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8424 PyUnicodeObject *result = NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8425 PyObject *dict = NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8426 PyObject *uformat;
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8427
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8428 if (format == NULL || args == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8429 PyErr_BadInternalCall();
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8430 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8431 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8432 uformat = PyUnicode_FromObject(format);
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
8433 if (uformat == NULL)
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
8434 return NULL;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8435 fmt = PyUnicode_AS_UNICODE(uformat);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8436 fmtcnt = PyUnicode_GET_SIZE(uformat);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8437
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8438 reslen = rescnt = fmtcnt + 100;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8439 result = _PyUnicode_New(reslen);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8440 if (result == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8441 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8442 res = PyUnicode_AS_UNICODE(result);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8443
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8444 if (PyTuple_Check(args)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8445 arglen = PyTuple_Size(args);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8446 argidx = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8447 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8448 else {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8449 arglen = -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8450 argidx = -2;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8451 }
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 43932
diff changeset
8452 if (Py_TYPE(args)->tp_as_mapping && !PyTuple_Check(args) &&
26164
63a08639dae2 Fix SF # 635969, No error "not all arguments converted"
Neal Norwitz <nnorwitz@gmail.com>
parents: 25895
diff changeset
8453 !PyObject_TypeCheck(args, &PyBaseString_Type))
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8454 dict = args;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8455
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8456 while (--fmtcnt >= 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8457 if (*fmt != '%') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8458 if (--rescnt < 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8459 rescnt = fmtcnt + 100;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8460 reslen += rescnt;
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
8461 if (_PyUnicode_Resize(&result, reslen) < 0)
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
8462 goto onError;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8463 res = PyUnicode_AS_UNICODE(result) + reslen - rescnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8464 --rescnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8465 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8466 *res++ = *fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8467 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8468 else {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8469 /* Got a format specifier */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8470 int flags = 0;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8471 Py_ssize_t width = -1;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8472 int prec = -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8473 Py_UNICODE c = '\0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8474 Py_UNICODE fill;
45500
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8475 int isnumok;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8476 PyObject *v = NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8477 PyObject *temp = NULL;
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8478 Py_UNICODE *pbuf;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8479 Py_UNICODE sign;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8480 Py_ssize_t len;
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8481 Py_UNICODE formatbuf[FORMATBUFLEN]; /* For format{float,int,char}() */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8482
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8483 fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8484 if (*fmt == '(') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8485 Py_UNICODE *keystart;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8486 Py_ssize_t keylen;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8487 PyObject *key;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8488 int pcount = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8489
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8490 if (dict == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8491 PyErr_SetString(PyExc_TypeError,
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8492 "format requires a mapping");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8493 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8494 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8495 ++fmt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8496 --fmtcnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8497 keystart = fmt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8498 /* Skip over balanced parentheses */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8499 while (pcount > 0 && --fmtcnt >= 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8500 if (*fmt == ')')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8501 --pcount;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8502 else if (*fmt == '(')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8503 ++pcount;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8504 fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8505 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8506 keylen = fmt - keystart - 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8507 if (fmtcnt < 0 || pcount > 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8508 PyErr_SetString(PyExc_ValueError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8509 "incomplete format key");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8510 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8511 }
21028
3dfbbf42bf16 Fix for bug #438164: %-formatting using Unicode objects.
Marc-André Lemburg <mal@egenix.com>
parents: 20563
diff changeset
8512 #if 0
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
8513 /* keys are converted to strings using UTF-8 and
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8514 then looked up since Python uses strings to hold
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8515 variables names etc. in its namespaces and we
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
8516 wouldn't want to break common idioms. */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8517 key = PyUnicode_EncodeUTF8(keystart,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8518 keylen,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8519 NULL);
21028
3dfbbf42bf16 Fix for bug #438164: %-formatting using Unicode objects.
Marc-André Lemburg <mal@egenix.com>
parents: 20563
diff changeset
8520 #else
3dfbbf42bf16 Fix for bug #438164: %-formatting using Unicode objects.
Marc-André Lemburg <mal@egenix.com>
parents: 20563
diff changeset
8521 key = PyUnicode_FromUnicode(keystart, keylen);
3dfbbf42bf16 Fix for bug #438164: %-formatting using Unicode objects.
Marc-André Lemburg <mal@egenix.com>
parents: 20563
diff changeset
8522 #endif
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8523 if (key == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8524 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8525 if (args_owned) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8526 Py_DECREF(args);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8527 args_owned = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8528 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8529 args = PyObject_GetItem(dict, key);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8530 Py_DECREF(key);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8531 if (args == NULL) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8532 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8533 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8534 args_owned = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8535 arglen = -1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8536 argidx = -2;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8537 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8538 while (--fmtcnt >= 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8539 switch (c = *fmt++) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8540 case '-': flags |= F_LJUST; continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8541 case '+': flags |= F_SIGN; continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8542 case ' ': flags |= F_BLANK; continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8543 case '#': flags |= F_ALT; continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8544 case '0': flags |= F_ZERO; continue;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8545 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8546 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8547 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8548 if (c == '*') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8549 v = getnextarg(args, arglen, &argidx);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8550 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8551 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8552 if (!PyInt_Check(v)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8553 PyErr_SetString(PyExc_TypeError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8554 "* wants int");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8555 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8556 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8557 width = PyInt_AsLong(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8558 if (width < 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8559 flags |= F_LJUST;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8560 width = -width;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8561 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8562 if (--fmtcnt >= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8563 c = *fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8564 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8565 else if (c >= '0' && c <= '9') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8566 width = c - '0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8567 while (--fmtcnt >= 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8568 c = *fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8569 if (c < '0' || c > '9')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8570 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8571 if ((width*10) / 10 != width) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8572 PyErr_SetString(PyExc_ValueError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8573 "width too big");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8574 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8575 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8576 width = width*10 + (c - '0');
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8577 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8578 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8579 if (c == '.') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8580 prec = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8581 if (--fmtcnt >= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8582 c = *fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8583 if (c == '*') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8584 v = getnextarg(args, arglen, &argidx);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8585 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8586 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8587 if (!PyInt_Check(v)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8588 PyErr_SetString(PyExc_TypeError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8589 "* wants int");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8590 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8591 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8592 prec = PyInt_AsLong(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8593 if (prec < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8594 prec = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8595 if (--fmtcnt >= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8596 c = *fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8597 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8598 else if (c >= '0' && c <= '9') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8599 prec = c - '0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8600 while (--fmtcnt >= 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8601 c = Py_CHARMASK(*fmt++);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8602 if (c < '0' || c > '9')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8603 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8604 if ((prec*10) / 10 != prec) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8605 PyErr_SetString(PyExc_ValueError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8606 "prec too big");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8607 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8608 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8609 prec = prec*10 + (c - '0');
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8610 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8611 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8612 } /* prec */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8613 if (fmtcnt >= 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8614 if (c == 'h' || c == 'l' || c == 'L') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8615 if (--fmtcnt >= 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8616 c = *fmt++;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8617 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8618 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8619 if (fmtcnt < 0) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8620 PyErr_SetString(PyExc_ValueError,
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8621 "incomplete format");
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8622 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8623 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8624 if (c != '%') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8625 v = getnextarg(args, arglen, &argidx);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8626 if (v == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8627 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8628 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8629 sign = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8630 fill = ' ';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8631 switch (c) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8632
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8633 case '%':
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8634 pbuf = formatbuf;
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8635 /* presume that buffer length is at least 1 */
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8636 pbuf[0] = '%';
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8637 len = 1;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8638 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8639
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8640 case 's':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8641 case 'r':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8642 if (PyUnicode_Check(v) && c == 's') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8643 temp = v;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8644 Py_INCREF(temp);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8645 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8646 else {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8647 PyObject *unicode;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8648 if (c == 's')
32801
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8649 temp = PyObject_Unicode(v);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8650 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8651 temp = PyObject_Repr(v);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8652 if (temp == NULL)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8653 goto onError;
32801
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8654 if (PyUnicode_Check(temp))
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8655 /* nothing to do */;
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8656 else if (PyString_Check(temp)) {
32801
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8657 /* convert to string to Unicode */
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8658 unicode = PyUnicode_Decode(PyString_AS_STRING(temp),
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47392
diff changeset
8659 PyString_GET_SIZE(temp),
36945
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8660 NULL,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8661 "strict");
36945
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8662 Py_DECREF(temp);
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8663 temp = unicode;
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8664 if (temp == NULL)
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8665 goto onError;
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8666 }
32801
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8667 else {
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8668 Py_DECREF(temp);
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8669 PyErr_SetString(PyExc_TypeError,
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8670 "%s argument has non-string str()");
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8671 goto onError;
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8672 }
16de4f90ef44 Let u'%s' % obj try obj.__unicode__() first and fallback to obj.__str__().
Marc-André Lemburg <mal@egenix.com>
parents: 32695
diff changeset
8673 }
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8674 pbuf = PyUnicode_AS_UNICODE(temp);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8675 len = PyUnicode_GET_SIZE(temp);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8676 if (prec >= 0 && len > prec)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8677 len = prec;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8678 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8679
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8680 case 'i':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8681 case 'd':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8682 case 'u':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8683 case 'o':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8684 case 'x':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8685 case 'X':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8686 if (c == 'i')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8687 c = 'd';
45500
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8688 isnumok = 0;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8689 if (PyNumber_Check(v)) {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8690 PyObject *iobj=NULL;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8691
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8692 if (PyInt_Check(v) || (PyLong_Check(v))) {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8693 iobj = v;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8694 Py_INCREF(iobj);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8695 }
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8696 else {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8697 iobj = PyNumber_Int(v);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8698 if (iobj==NULL) iobj = PyNumber_Long(v);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8699 }
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8700 if (iobj!=NULL) {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8701 if (PyInt_Check(iobj)) {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8702 isnumok = 1;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8703 pbuf = formatbuf;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8704 len = formatint(pbuf, sizeof(formatbuf)/sizeof(Py_UNICODE),
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8705 flags, prec, c, iobj);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8706 Py_DECREF(iobj);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8707 if (len < 0)
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8708 goto onError;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8709 sign = 1;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8710 }
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8711 else if (PyLong_Check(iobj)) {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8712 isnumok = 1;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8713 temp = formatlong(iobj, flags, prec, c);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8714 Py_DECREF(iobj);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8715 if (!temp)
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8716 goto onError;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8717 pbuf = PyUnicode_AS_UNICODE(temp);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8718 len = PyUnicode_GET_SIZE(temp);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8719 sign = 1;
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8720 }
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8721 else {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8722 Py_DECREF(iobj);
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8723 }
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8724 }
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8725 }
45500
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8726 if (!isnumok) {
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8727 PyErr_Format(PyExc_TypeError,
c03aff6d07f5 Issue 1742669. Now %d accepts very big float numbers.
Facundo Batista <facundobatista@gmail.com>
parents: 45381
diff changeset
8728 "%%%c format: a number is required, "
46333
4e72489f16d0 Bug #2388: Fix gcc warnings when compiling with --enable-unicode=ucs4.
Martin v. Löwis <martin@v.loewis.de>
parents: 46149
diff changeset
8729 "not %.200s", (char)c, Py_TYPE(v)->tp_name);
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8730 goto onError;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8731 }
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8732 if (flags & F_ZERO)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8733 fill = '0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8734 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8735
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8736 case 'e':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8737 case 'E':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8738 case 'f':
30216
7ab1a41f16da SF bug #795506: Wrong handling of string format code for float values.
Raymond Hettinger <python@rcn.com>
parents: 30164
diff changeset
8739 case 'F':
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8740 case 'g':
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8741 case 'G':
48003
5c33009994b4 Backed out r65069, pending fixing it in Windows.
Eric Smith <eric@trueblade.com>
parents: 48001
diff changeset
8742 if (c == 'F')
5c33009994b4 Backed out r65069, pending fixing it in Windows.
Eric Smith <eric@trueblade.com>
parents: 48001
diff changeset
8743 c = 'f';
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8744 pbuf = formatbuf;
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8745 len = formatfloat(pbuf, sizeof(formatbuf)/sizeof(Py_UNICODE),
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8746 flags, prec, c, v);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8747 if (len < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8748 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8749 sign = 1;
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8750 if (flags & F_ZERO)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8751 fill = '0';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8752 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8753
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8754 case 'c':
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8755 pbuf = formatbuf;
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8756 len = formatchar(pbuf, sizeof(formatbuf)/sizeof(Py_UNICODE), v);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8757 if (len < 0)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8758 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8759 break;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8760
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8761 default:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8762 PyErr_Format(PyExc_ValueError,
15579
31d78f361530 [ Patch #102852 ] Make % error a bit more informative by indicates the
Andrew M. Kuchling <amk@amk.ca>
parents: 15455
diff changeset
8763 "unsupported format character '%c' (0x%x) "
39995
6ba402b60041 Forward-port of r52136,52138: a review of overflow-detecting code.
Armin Rigo <arigo@tunes.org>
parents: 39734
diff changeset
8764 "at index %zd",
30441
76e998aee644 On c.l.py, Martin v. Löwis said that Py_UNICODE could be of a signed type,
Tim Peters <tim.peters@gmail.com>
parents: 30439
diff changeset
8765 (31<=c && c<=126) ? (char)c : '?',
25375
4df0e691f4da Add cast to avoid compiler warning.
Marc-André Lemburg <mal@egenix.com>
parents: 25248
diff changeset
8766 (int)c,
39995
6ba402b60041 Forward-port of r52136,52138: a review of overflow-detecting code.
Armin Rigo <arigo@tunes.org>
parents: 39734
diff changeset
8767 (Py_ssize_t)(fmt - 1 -
6ba402b60041 Forward-port of r52136,52138: a review of overflow-detecting code.
Armin Rigo <arigo@tunes.org>
parents: 39734
diff changeset
8768 PyUnicode_AS_UNICODE(uformat)));
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8769 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8770 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8771 if (sign) {
13196
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8772 if (*pbuf == '-' || *pbuf == '+') {
18920eaa1226 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 13110
diff changeset
8773 sign = *pbuf++;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8774 len--;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8775 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8776 else if (flags & F_SIGN)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8777 sign = '+';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8778 else if (flags & F_BLANK)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8779 sign = ' ';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8780 else
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8781 sign = 0;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8782 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8783 if (width < len)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8784 width = len;
25719
f532e9b69551 Fix a nasty endcase reported by Armin Rigo in SF bug 618623:
Guido van Rossum <guido@python.org>
parents: 25375
diff changeset
8785 if (rescnt - (sign != 0) < width) {
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8786 reslen -= rescnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8787 rescnt = width + fmtcnt + 100;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8788 reslen += rescnt;
25719
f532e9b69551 Fix a nasty endcase reported by Armin Rigo in SF bug 618623:
Guido van Rossum <guido@python.org>
parents: 25375
diff changeset
8789 if (reslen < 0) {
36877
e9a7a2784e17 SF #1444030: Fix several potential defects found by Coverity.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36675
diff changeset
8790 Py_XDECREF(temp);
36945
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8791 PyErr_NoMemory();
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8792 goto onError;
25719
f532e9b69551 Fix a nasty endcase reported by Armin Rigo in SF bug 618623:
Guido van Rossum <guido@python.org>
parents: 25375
diff changeset
8793 }
36945
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8794 if (_PyUnicode_Resize(&result, reslen) < 0) {
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8795 Py_XDECREF(temp);
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8796 goto onError;
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8797 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8798 res = PyUnicode_AS_UNICODE(result)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8799 + reslen - rescnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8800 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8801 if (sign) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8802 if (fill != ' ')
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8803 *res++ = sign;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8804 rescnt--;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8805 if (width > len)
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8806 width--;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8807 }
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8808 if ((flags & F_ALT) && (c == 'x' || c == 'X')) {
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8809 assert(pbuf[0] == '0');
17450
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8810 assert(pbuf[1] == c);
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8811 if (fill != ' ') {
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8812 *res++ = *pbuf++;
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8813 *res++ = *pbuf++;
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8814 }
17450
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8815 rescnt -= 2;
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8816 width -= 2;
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8817 if (width < 0)
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8818 width = 0;
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8819 len -= 2;
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8820 }
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8821 if (width > len && !(flags & F_LJUST)) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8822 do {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8823 --rescnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8824 *res++ = fill;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8825 } while (--width > len);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8826 }
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8827 if (fill == ' ') {
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8828 if (sign)
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8829 *res++ = sign;
17450
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8830 if ((flags & F_ALT) && (c == 'x' || c == 'X')) {
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8831 assert(pbuf[0] == '0');
17450
f74f5aafdc7e Bug 415514 reported that e.g.
Tim Peters <tim.peters@gmail.com>
parents: 17425
diff changeset
8832 assert(pbuf[1] == c);
14713
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8833 *res++ = *pbuf++;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8834 *res++ = *pbuf++;
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8835 }
58b832b8745e Derived from Martin's SF patch 110609: support unbounded ints in %d,i,u,x,X,o formats.
Tim Peters <tim.peters@gmail.com>
parents: 14492
diff changeset
8836 }
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
8837 Py_UNICODE_COPY(res, pbuf, len);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8838 res += len;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8839 rescnt -= len;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8840 while (--width >= len) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8841 --rescnt;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8842 *res++ = ' ';
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8843 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8844 if (dict && (argidx < arglen) && c != '%') {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8845 PyErr_SetString(PyExc_TypeError,
23494
76f376bea5e5 Patch 549187. Improve string formatting error message.
Raymond Hettinger <python@rcn.com>
parents: 23295
diff changeset
8846 "not all arguments converted during string formatting");
36945
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8847 Py_XDECREF(temp);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8848 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8849 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8850 Py_XDECREF(temp);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8851 } /* '%' */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8852 } /* until end */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8853 if (argidx < arglen && !dict) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8854 PyErr_SetString(PyExc_TypeError,
23494
76f376bea5e5 Patch 549187. Improve string formatting error message.
Raymond Hettinger <python@rcn.com>
parents: 23295
diff changeset
8855 "not all arguments converted during string formatting");
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8856 goto onError;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8857 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8858
36945
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8859 if (_PyUnicode_Resize(&result, reslen - rescnt) < 0)
0f19a6fc9756 - Reindent a confusingly indented piece of code (no intended code changes
Thomas Wouters <thomas@python.org>
parents: 36924
diff changeset
8860 goto onError;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8861 if (args_owned) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8862 Py_DECREF(args);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8863 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8864 Py_DECREF(uformat);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8865 return (PyObject *)result;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8866
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8867 onError:
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8868 Py_XDECREF(result);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8869 Py_DECREF(uformat);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8870 if (args_owned) {
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8871 Py_DECREF(args);
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8872 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8873 return NULL;
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8874 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8875
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8876 static PyBufferProcs unicode_as_buffer = {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8877 (readbufferproc) unicode_buffer_getreadbuf,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8878 (writebufferproc) unicode_buffer_getwritebuf,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8879 (segcountproc) unicode_buffer_getsegcount,
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8880 (charbufferproc) unicode_buffer_getcharbuf,
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8881 };
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8882
24280
1598a1e473e2 staticforward bites the dust.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 24071
diff changeset
8883 static PyObject *
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8884 unicode_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds);
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8885
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8886 static PyObject *
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8887 unicode_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8888 {
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8889 PyObject *x = NULL;
36675
dba6494735d0 Revert backwards-incompatible const changes.
Martin v. Löwis <martin@v.loewis.de>
parents: 36532
diff changeset
8890 static char *kwlist[] = {"string", "encoding", "errors", 0};
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8891 char *encoding = NULL;
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8892 char *errors = NULL;
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8893
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8894 if (type != &PyUnicode_Type)
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8895 return unicode_subtype_new(type, args, kwds);
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8896 if (!PyArg_ParseTupleAndKeywords(args, kwds, "|Oss:unicode",
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8897 kwlist, &x, &encoding, &errors))
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8898 return NULL;
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8899 if (x == NULL)
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8900 return (PyObject *)_PyUnicode_New(0);
20554
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
8901 if (encoding == NULL && errors == NULL)
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
8902 return PyObject_Unicode(x);
0336ee5b46d4 SF patch #470578: Fixes to synchronize unicode() and str()
Guido van Rossum <guido@python.org>
parents: 20277
diff changeset
8903 else
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8904 return PyUnicode_FromEncodedObject(x, encoding, errors);
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8905 }
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8906
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8907 static PyObject *
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8908 unicode_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8909 {
19886
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8910 PyUnicodeObject *tmp, *pnew;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36261
diff changeset
8911 Py_ssize_t n;
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8912
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8913 assert(PyType_IsSubtype(type, &PyUnicode_Type));
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8914 tmp = (PyUnicodeObject *)unicode_new(&PyUnicode_Type, args, kwds);
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8915 if (tmp == NULL)
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8916 return NULL;
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8917 assert(PyUnicode_Check(tmp));
19886
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8918 pnew = (PyUnicodeObject *) type->tp_alloc(type, n = tmp->length);
29641
4909aa64405f SF patch 703666: Several objects don't decref tmp on failure in subtype_new
Raymond Hettinger <python@rcn.com>
parents: 29164
diff changeset
8919 if (pnew == NULL) {
4909aa64405f SF patch 703666: Several objects don't decref tmp on failure in subtype_new
Raymond Hettinger <python@rcn.com>
parents: 29164
diff changeset
8920 Py_DECREF(tmp);
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8921 return NULL;
29641
4909aa64405f SF patch 703666: Several objects don't decref tmp on failure in subtype_new
Raymond Hettinger <python@rcn.com>
parents: 29164
diff changeset
8922 }
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
8923 pnew->str = (Py_UNICODE*) PyObject_MALLOC(sizeof(Py_UNICODE) * (n+1));
19886
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8924 if (pnew->str == NULL) {
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8925 _Py_ForgetReference((PyObject *)pnew);
22981
dc1fb19d990d Remove PyMalloc_*.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 22938
diff changeset
8926 PyObject_Del(pnew);
29641
4909aa64405f SF patch 703666: Several objects don't decref tmp on failure in subtype_new
Raymond Hettinger <python@rcn.com>
parents: 29164
diff changeset
8927 Py_DECREF(tmp);
27826
7dffcc498c45 Add more missing PyErr_NoMemory() after failled memory allocs
Neal Norwitz <nnorwitz@gmail.com>
parents: 27774
diff changeset
8928 return PyErr_NoMemory();
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8929 }
19886
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8930 Py_UNICODE_COPY(pnew->str, tmp->str, n+1);
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8931 pnew->length = n;
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8932 pnew->hash = tmp->hash;
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8933 Py_DECREF(tmp);
19886
3b0da13ace48 str_subtype_new, unicode_subtype_new:
Tim Peters <tim.peters@gmail.com>
parents: 19885
diff changeset
8934 return (PyObject *)pnew;
19593
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8935 }
a80b4b006eb7 Make unicode subclassable.
Guido van Rossum <guido@python.org>
parents: 19301
diff changeset
8936
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23778
diff changeset
8937 PyDoc_STRVAR(unicode_doc,
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8938 "unicode(string [, encoding[, errors]]) -> object\n\
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8939 \n\
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8940 Create a new Unicode object from the given encoded string.\n\
24392
0a327b768868 tighten up the unicode object's docstring a tad
Skip Montanaro <skip@pobox.com>
parents: 24280
diff changeset
8941 encoding defaults to the current default string encoding.\n\
0a327b768868 tighten up the unicode object's docstring a tad
Skip Montanaro <skip@pobox.com>
parents: 24280
diff changeset
8942 errors can be 'strict', 'replace' or 'ignore' and defaults to 'strict'.");
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8943
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8944 PyTypeObject PyUnicode_Type = {
42333
6620a510e9a9 PEP 3123: Provide forward compatibility with Python 3.0, while keeping
Martin v. Löwis <martin@v.loewis.de>
parents: 42216
diff changeset
8945 PyVarObject_HEAD_INIT(&PyType_Type, 0)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8946 "unicode", /* tp_name */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8947 sizeof(PyUnicodeObject), /* tp_size */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8948 0, /* tp_itemsize */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8949 /* Slots */
20277
2983b54256c7 Enable GC for new-style instances. This touches lots of files, since
Guido van Rossum <guido@python.org>
parents: 20018
diff changeset
8950 (destructor)unicode_dealloc, /* tp_dealloc */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8951 0, /* tp_print */
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8952 0, /* tp_getattr */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8953 0, /* tp_setattr */
39487
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
8954 0, /* tp_compare */
37304
79a0f080f898 Remove unnecessary casts in type object initializers.
Georg Brandl <georg@python.org>
parents: 36945
diff changeset
8955 unicode_repr, /* tp_repr */
26274
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8956 &unicode_as_number, /* tp_as_number */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8957 &unicode_as_sequence, /* tp_as_sequence */
23778
245224d1b8c9 This is my nearly two year old patch
Michael W. Hudson <mwh@python.net>
parents: 23562
diff changeset
8958 &unicode_as_mapping, /* tp_as_mapping */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8959 (hashfunc) unicode_hash, /* tp_hash*/
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8960 0, /* tp_call*/
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8961 (reprfunc) unicode_str, /* tp_str */
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8962 PyObject_GenericGetAttr, /* tp_getattro */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8963 0, /* tp_setattro */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8964 &unicode_as_buffer, /* tp_as_buffer */
26274
958d07085dcf Add nb_remainder (i.e. __mod__) slot to unicode type. Fixes SF bug #615506.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 26164
diff changeset
8965 Py_TPFLAGS_DEFAULT | Py_TPFLAGS_CHECKTYPES |
40908
d01bd37bf5ce Variation of patch # 1624059 to speed up checking if an object is a subclass
Neal Norwitz <nnorwitz@gmail.com>
parents: 39995
diff changeset
8966 Py_TPFLAGS_BASETYPE | Py_TPFLAGS_UNICODE_SUBCLASS, /* tp_flags */
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8967 unicode_doc, /* tp_doc */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8968 0, /* tp_traverse */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8969 0, /* tp_clear */
39487
635b09b1f542 Slightly revised version of patch #1538956:
Marc-André Lemburg <mal@egenix.com>
parents: 39463
diff changeset
8970 PyUnicode_RichCompare, /* tp_richcompare */
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8971 0, /* tp_weaklistoffset */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8972 0, /* tp_iter */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8973 0, /* tp_iternext */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8974 unicode_methods, /* tp_methods */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8975 0, /* tp_members */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8976 0, /* tp_getset */
23542
220b98f06415 - A new type object, 'string', is added. This is a common base type
Guido van Rossum <guido@python.org>
parents: 23494
diff changeset
8977 &PyBaseString_Type, /* tp_base */
18933
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8978 0, /* tp_dict */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8979 0, /* tp_descr_get */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8980 0, /* tp_descr_set */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8981 0, /* tp_dictoffset */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8982 0, /* tp_init */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8983 0, /* tp_alloc */
09df3254b49d Merge of descr-branch back into trunk.
Tim Peters <tim.peters@gmail.com>
parents: 18887
diff changeset
8984 unicode_new, /* tp_new */
22981
dc1fb19d990d Remove PyMalloc_*.
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 22938
diff changeset
8985 PyObject_Del, /* tp_free */
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8986 };
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8987
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8988 /* Initialize the Unicode implementation */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8989
13660
a1ce07218781 Miscelaneous ANSIfications. I'm assuming here 'main' should take (int,
Thomas Wouters <thomas@python.org>
parents: 13635
diff changeset
8990 void _PyUnicode_Init(void)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8991 {
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
8992 int i;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
8993
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
8994 /* XXX - move this array to unicodectype.c ? */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
8995 Py_UNICODE linebreak[] = {
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
8996 0x000A, /* LINE FEED */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
8997 0x000D, /* CARRIAGE RETURN */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
8998 0x001C, /* FILE SEPARATOR */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
8999 0x001D, /* GROUP SEPARATOR */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9000 0x001E, /* RECORD SEPARATOR */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9001 0x0085, /* NEXT LINE */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9002 0x2028, /* LINE SEPARATOR */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9003 0x2029, /* PARAGRAPH SEPARATOR */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9004 };
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9005
12645
c4eb70d24a87 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12629
diff changeset
9006 /* Init the implementation */
45191
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
9007 free_list = NULL;
920a7633ae08 Unified naming convention for free lists and their limits. All free lists
Christian Heimes <christian@cheimes.de>
parents: 45053
diff changeset
9008 numfree = 0;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9009 unicode_empty = _PyUnicode_New(0);
39129
f2031f15d99d Handle allocation failures gracefully. Found with failmalloc.
Neal Norwitz <nnorwitz@gmail.com>
parents: 38817
diff changeset
9010 if (!unicode_empty)
f2031f15d99d Handle allocation failures gracefully. Found with failmalloc.
Neal Norwitz <nnorwitz@gmail.com>
parents: 38817
diff changeset
9011 return;
f2031f15d99d Handle allocation failures gracefully. Found with failmalloc.
Neal Norwitz <nnorwitz@gmail.com>
parents: 38817
diff changeset
9012
12876
243593f11fcf Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12646
diff changeset
9013 strcpy(unicode_default_encoding, "ascii");
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9014 for (i = 0; i < 256; i++)
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9015 unicode_latin1[i] = NULL;
23542
220b98f06415 - A new type object, 'string', is added. This is a common base type
Guido van Rossum <guido@python.org>
parents: 23494
diff changeset
9016 if (PyType_Ready(&PyUnicode_Type) < 0)
220b98f06415 - A new type object, 'string', is added. This is a common base type
Guido van Rossum <guido@python.org>
parents: 23494
diff changeset
9017 Py_FatalError("Can't initialize 'unicode'");
38257
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9018
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9019 /* initialize the linebreak bloom filter */
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9020 bloom_linebreak = make_bloom_mask(
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9021 linebreak, sizeof(linebreak) / sizeof(linebreak[0])
af46c7ed05eb needforspeed: use append+reverse for rsplit, use "bloom filters" to
Fredrik Lundh <fredrik@pythonware.com>
parents: 38243
diff changeset
9022 );
38786
9c52efcaff65 Initialize the type object so pychecker can't crash the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 38709
diff changeset
9023
9c52efcaff65 Initialize the type object so pychecker can't crash the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 38709
diff changeset
9024 PyType_Ready(&EncodingMapType);
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9025 }
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9026
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9027 /* Finalize the Unicode implementation */
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9028
45341
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9029 int
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9030 PyUnicode_ClearFreeList(void)
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9031 {
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9032 int freelist_size = numfree;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9033 PyUnicodeObject *u;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9034
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9035 for (u = free_list; u != NULL;) {
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9036 PyUnicodeObject *v = u;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9037 u = *(PyUnicodeObject **)u;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9038 if (v->str)
45804
d33cadb595f6 Issue 2321: reduce memory usage (increase the memory that is returned
Neal Norwitz <nnorwitz@gmail.com>
parents: 45731
diff changeset
9039 PyObject_DEL(v->str);
45341
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9040 Py_XDECREF(v->defenc);
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9041 PyObject_Del(v);
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9042 numfree--;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9043 }
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9044 free_list = NULL;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9045 assert(numfree == 0);
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9046 return freelist_size;
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9047 }
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9048
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9049 void
13660
a1ce07218781 Miscelaneous ANSIfications. I'm assuming here 'main' should take (int,
Thomas Wouters <thomas@python.org>
parents: 13635
diff changeset
9050 _PyUnicode_Fini(void)
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9051 {
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9052 int i;
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9053
14998
c16bdf555c79 In _PyUnicode_Fini(), decref unicode_empty before tearng down the free
Guido van Rossum <guido@python.org>
parents: 14869
diff changeset
9054 Py_XDECREF(unicode_empty);
c16bdf555c79 In _PyUnicode_Fini(), decref unicode_empty before tearng down the free
Guido van Rossum <guido@python.org>
parents: 14869
diff changeset
9055 unicode_empty = NULL;
15001
a349b89bf10e _PyUnicode_Fini(): Initialize the local freelist walking variable `u'
Barry Warsaw <barry@python.org>
parents: 14998
diff changeset
9056
17618
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9057 for (i = 0; i < 256; i++) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9058 if (unicode_latin1[i]) {
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9059 Py_DECREF(unicode_latin1[i]);
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9060 unicode_latin1[i] = NULL;
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9061 }
b44ba7c01217 This patch originated from an idea by Martin v. Loewis who submitted a
Marc-André Lemburg <mal@egenix.com>
parents: 17600
diff changeset
9062 }
45341
65ae68c7aca3 Implemented Martin's suggestion to clear the free lists during the garbage collection of the highest generation.
Christian Heimes <christian@cheimes.de>
parents: 45191
diff changeset
9063 (void)PyUnicode_ClearFreeList();
11968
13bc68faeb65 Unicode implementation by Marc-Andre Lemburg based on original code by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9064 }
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9065
37655
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
9066 #ifdef __cplusplus
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
9067 }
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
9068 #endif
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
9069
2f3f2e32dfd9 spread the extern "C" { } magic pixie dust around. Python itself builds now
Anthony Baxter <anthonybaxter@gmail.com>
parents: 37596
diff changeset
9070
29164
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9071 /*
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9072 Local variables:
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9073 c-basic-offset: 4
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9074 indent-tabs-mode: nil
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9075 End:
4d9017ef7b5f Consider \U-escapes in raw-unicode-escape. Fixes #444514.
Martin v. Löwis <martin@v.loewis.de>
parents: 28575
diff changeset
9076 */