Mercurial > cpython-lru_cache-weakref

annotate Modules/unicodedata.c @ 58765:2ba7677bfa95 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
testCopysign was defined twice in test_math; combine the definitions
author Mark Dickinson <dickinsm@gmail.com>
date Sat, 06 Feb 2010 23:11:25 +0000
parents a9864417286b
children 26269dfc22ab
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 /* ------------------------------------------------------------------------
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
3 unicodedata -- Provides access to the Unicode 5.1 data base.
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
4
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
5 Data was extracted from the Unicode 5.1 UnicodeData.txt file.
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14818
diff changeset
7 Written by Marc-Andre Lemburg (mal@lemburg.com).
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14818
diff changeset
8 Modified for Python 2.0 by Fredrik Lundh (fredrik@pythonware.com)
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
9 Modified by Martin v. Löwis (martin@v.loewis.de)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14818
diff changeset
11 Copyright (c) Corporation for National Research Initiatives.
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
12
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
13 ------------------------------------------------------------------------ */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
14
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
15 #include "Python.h"
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
16 #include "ucnhash.h"
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
17 #include "structmember.h"
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
18
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
19 /* character properties */
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
20
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
21 typedef struct {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
22 const unsigned char category; /* index into
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
23 _PyUnicode_CategoryNames */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
24 const unsigned char combining; /* combining class value 0 - 255 */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
25 const unsigned char bidirectional; /* index into
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
26 _PyUnicode_BidirectionalNames */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
27 const unsigned char mirrored; /* true if mirrored in bidir mode */
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
28 const unsigned char east_asian_width; /* index into
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
29 _PyUnicode_EastAsianWidth */
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
30 const unsigned char normalization_quick_check; /* see is_normalized() */
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
31 } _PyUnicode_DatabaseRecord;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
32
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
33 typedef struct change_record {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
34 /* sequence of fields should be the same as in merge_old_version */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
35 const unsigned char bidir_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
36 const unsigned char category_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
37 const unsigned char decimal_changed;
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
38 const unsigned char mirrored_changed;
56406
a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric,
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53553
diff changeset
39 const double numeric_changed;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
40 } change_record;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
41
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
42 /* data file generated by Tools/unicode/makeunicodedata.py */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
43 #include "unicodedata_db.h"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
44
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
45 static const _PyUnicode_DatabaseRecord*
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
46 _getrecord_ex(Py_UCS4 code)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
47 {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
48 int index;
28100
54b35da9879e Fix SF bug #694816, remove comparison of unsigned value < 0
Neal Norwitz <nnorwitz@gmail.com>
parents: 26571
diff changeset
49 if (code >= 0x110000)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
50 index = 0;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
51 else {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
52 index = index1[(code>>SHIFT)];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
53 index = index2[(index<<SHIFT)+(code&((1<<SHIFT)-1))];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
54 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
55
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
56 return &_PyUnicode_Database_Records[index];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
57 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
58
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
59 /* ------------- Previous-version API ------------------------------------- */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
60 typedef struct previous_version {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
61 PyObject_HEAD
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
62 const char *name;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
63 const change_record* (*getrecord)(Py_UCS4);
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
64 Py_UCS4 (*normalization)(Py_UCS4);
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
65 } PreviousDBVersion;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
66
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
67 #define get_old_record(self, v) ((((PreviousDBVersion*)self)->getrecord)(v))
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
68
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
69 static PyMemberDef DB_members[] = {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
70 {"unidata_version", T_STRING, offsetof(PreviousDBVersion, name), READONLY},
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
71 {NULL}
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
72 };
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
73
40319
7059df7c15df Replace C++ comment with C comment (fixes SF bug #1593525).
Walter Dörwald <walter@livinglogic.de>
parents: 39453
diff changeset
74 /* forward declaration */
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
75 static PyTypeObject UCD_Type;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
76
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
77 static PyObject*
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
78 new_previous_version(const char*name, const change_record* (*getrecord)(Py_UCS4),
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
79 Py_UCS4 (*normalization)(Py_UCS4))
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
80 {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
81 PreviousDBVersion *self;
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
82 self = PyObject_New(PreviousDBVersion, &UCD_Type);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
83 if (self == NULL)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
84 return NULL;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
85 self->name = name;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
86 self->getrecord = getrecord;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
87 self->normalization = normalization;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
88 return (PyObject*)self;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
89 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
90
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
91
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
92 static Py_UCS4 getuchar(PyUnicodeObject *obj)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
93 {
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
94 Py_UNICODE *v = PyUnicode_AS_UNICODE(obj);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
95
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
96 if (PyUnicode_GET_SIZE(obj) == 1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
97 return *v;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
98 #ifndef Py_UNICODE_WIDE
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
99 else if ((PyUnicode_GET_SIZE(obj) == 2) &&
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
100 (0xD800 <= v[0] && v[0] <= 0xDBFF) &&
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
101 (0xDC00 <= v[1] && v[1] <= 0xDFFF))
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
102 return (((v[0] & 0x3FF)<<10) | (v[1] & 0x3FF)) + 0x10000;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
103 #endif
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
104 PyErr_SetString(PyExc_TypeError,
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
105 "need a single Unicode character as parameter");
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
106 return (Py_UCS4)-1;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
107 }
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
108
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
109 /* --- Module API --------------------------------------------------------- */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
110
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
111 PyDoc_STRVAR(unicodedata_decimal__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
112 "decimal(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
113 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
114 Returns the decimal value assigned to the Unicode character unichr\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
115 as integer. If no such value is defined, default is returned, or, if\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
116 not given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
117
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
118 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
119 unicodedata_decimal(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
120 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122 PyObject *defobj = NULL;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
123 int have_old = 0;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 long rc;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
125 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
127 if (!PyArg_ParseTuple(args, "O!|O:decimal", &PyUnicode_Type, &v, &defobj))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
128 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
129 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
130 if (c == (Py_UCS4)-1)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
131 return NULL;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
132
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
133 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
134 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
135 if (old->category_changed == 0) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
136 /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
137 have_old = 1;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
138 rc = -1;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
139 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
140 else if (old->decimal_changed != 0xFF) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
141 have_old = 1;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
142 rc = old->decimal_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
143 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
144 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
145
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
146 if (!have_old)
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
147 rc = Py_UNICODE_TODECIMAL(c);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 if (rc < 0) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
149 if (defobj == NULL) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
150 PyErr_SetString(PyExc_ValueError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
151 "not a decimal");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
152 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
153 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
154 else {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
155 Py_INCREF(defobj);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
156 return defobj;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
157 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
158 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
159 return PyInt_FromLong(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
160 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
161
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
162 PyDoc_STRVAR(unicodedata_digit__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
163 "digit(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
164 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
165 Returns the digit value assigned to the Unicode character unichr as\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
166 integer. If no such value is defined, default is returned, or, if\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
167 not given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
168
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
169 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
170 unicodedata_digit(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
171 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
172 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 PyObject *defobj = NULL;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
174 long rc;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
175 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
176
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
177 if (!PyArg_ParseTuple(args, "O!|O:digit", &PyUnicode_Type, &v, &defobj))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
178 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
179 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
180 if (c == (Py_UCS4)-1)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
181 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
182 rc = Py_UNICODE_TODIGIT(c);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 if (rc < 0) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 if (defobj == NULL) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
185 PyErr_SetString(PyExc_ValueError, "not a digit");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
186 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
187 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
188 else {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
189 Py_INCREF(defobj);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
190 return defobj;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
191 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
192 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
193 return PyInt_FromLong(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
194 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
195
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
196 PyDoc_STRVAR(unicodedata_numeric__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
197 "numeric(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
198 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
199 Returns the numeric value assigned to the Unicode character unichr\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
200 as float. If no such value is defined, default is returned, or, if\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
201 not given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
202
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
203 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
204 unicodedata_numeric(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
207 PyObject *defobj = NULL;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
208 int have_old = 0;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 double rc;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
210 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
211
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
212 if (!PyArg_ParseTuple(args, "O!|O:numeric", &PyUnicode_Type, &v, &defobj))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
213 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
214 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
215 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
216 return NULL;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
217
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
218 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
219 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
220 if (old->category_changed == 0) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
221 /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
222 have_old = 1;
38422
dfbd457ca67a Patch 1494554: Update numeric properties to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 37780
diff changeset
223 rc = -1.0;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
224 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
225 else if (old->decimal_changed != 0xFF) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
226 have_old = 1;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
227 rc = old->decimal_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
228 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
229 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
230
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
231 if (!have_old)
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
232 rc = Py_UNICODE_TONUMERIC(c);
38422
dfbd457ca67a Patch 1494554: Update numeric properties to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 37780
diff changeset
233 if (rc == -1.0) {
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
234 if (defobj == NULL) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
235 PyErr_SetString(PyExc_ValueError, "not a numeric character");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
236 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
237 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
238 else {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
239 Py_INCREF(defobj);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
240 return defobj;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
241 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
242 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
243 return PyFloat_FromDouble(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
244 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
245
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
246 PyDoc_STRVAR(unicodedata_category__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
247 "category(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
248 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
249 Returns the general category assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
250 unichr as string.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
251
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
253 unicodedata_category(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
254 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
255 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
256 int index;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
257 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
258
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
259 if (!PyArg_ParseTuple(args, "O!:category",
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
260 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
261 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
262 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
263 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
264 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
265 index = (int) _getrecord_ex(c)->category;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
266 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
267 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
268 if (old->category_changed != 0xFF)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
269 index = old->category_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
270 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47451
diff changeset
271 return PyString_FromString(_PyUnicode_CategoryNames[index]);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
272 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
273
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
274 PyDoc_STRVAR(unicodedata_bidirectional__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
275 "bidirectional(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
276 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
277 Returns the bidirectional category assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
278 unichr as string. If no such value is defined, an empty string is\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
279 returned.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
280
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
281 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
282 unicodedata_bidirectional(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
283 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
284 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
285 int index;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
286 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
287
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
288 if (!PyArg_ParseTuple(args, "O!:bidirectional",
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
289 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
290 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
291 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
292 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
293 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
294 index = (int) _getrecord_ex(c)->bidirectional;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
295 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
296 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
297 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
298 index = 0; /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
299 else if (old->bidir_changed != 0xFF)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
300 index = old->bidir_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
301 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47451
diff changeset
302 return PyString_FromString(_PyUnicode_BidirectionalNames[index]);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
303 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
304
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
305 PyDoc_STRVAR(unicodedata_combining__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
306 "combining(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
307 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
308 Returns the canonical combining class assigned to the Unicode\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
309 character unichr as integer. Returns 0 if no combining class is\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
310 defined.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
311
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
312 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
313 unicodedata_combining(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
314 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
315 PyUnicodeObject *v;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
316 int index;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
317 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
318
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
319 if (!PyArg_ParseTuple(args, "O!:combining",
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
320 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
321 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
322 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
323 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
324 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
325 index = (int) _getrecord_ex(c)->combining;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
326 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
327 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
328 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
329 index = 0; /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
330 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
331 return PyInt_FromLong(index);
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
332 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
333
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
334 PyDoc_STRVAR(unicodedata_mirrored__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
335 "mirrored(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
336 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
337 Returns the mirrored property assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
338 unichr as integer. Returns 1 if the character has been identified as\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
339 a \"mirrored\" character in bidirectional text, 0 otherwise.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
340
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
341 static PyObject *
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
342 unicodedata_mirrored(PyObject *self, PyObject *args)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
343 {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
344 PyUnicodeObject *v;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
345 int index;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
346 Py_UCS4 c;
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
347
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
348 if (!PyArg_ParseTuple(args, "O!:mirrored",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
349 &PyUnicode_Type, &v))
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
350 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
351 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
352 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
353 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
354 index = (int) _getrecord_ex(c)->mirrored;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
355 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
356 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
357 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
358 index = 0; /* unassigned */
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
359 else if (old->mirrored_changed != 0xFF)
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
360 index = old->mirrored_changed;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
361 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
362 return PyInt_FromLong(index);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
363 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
364
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
365 PyDoc_STRVAR(unicodedata_east_asian_width__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
366 "east_asian_width(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
367 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
368 Returns the east asian width assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
369 unichr as string.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
370
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
371 static PyObject *
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
372 unicodedata_east_asian_width(PyObject *self, PyObject *args)
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
373 {
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
374 PyUnicodeObject *v;
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
375 int index;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
376 Py_UCS4 c;
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
377
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
378 if (!PyArg_ParseTuple(args, "O!:east_asian_width",
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
379 &PyUnicode_Type, &v))
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
380 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
381 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
382 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
383 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
384 index = (int) _getrecord_ex(c)->east_asian_width;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
385 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
386 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
387 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
388 index = 0; /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
389 }
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47451
diff changeset
390 return PyString_FromString(_PyUnicode_EastAsianWidthNames[index]);
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
391 }
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
392
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
393 PyDoc_STRVAR(unicodedata_decomposition__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
394 "decomposition(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
395 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
396 Returns the character decomposition mapping assigned to the Unicode\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
397 character unichr as string. An empty string is returned in case no\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
398 such mapping is defined.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
399
32951
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32680
diff changeset
400 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
401 unicodedata_decomposition(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
402 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
403 PyUnicodeObject *v;
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
404 char decomp[256];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
405 int code, index, count, i;
39203
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
406 unsigned int prefix_index;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
407 Py_UCS4 c;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
408
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
409 if (!PyArg_ParseTuple(args, "O!:decomposition",
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
410 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
411 return NULL;
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
412 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
413 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
414 return NULL;
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
415
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
416 code = (int)c;
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
417
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
418 if (self) {
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
419 const change_record *old = get_old_record(self, c);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
420 if (old->category_changed == 0)
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47451
diff changeset
421 return PyString_FromString(""); /* unassigned */
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
422 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
423
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 24363
diff changeset
424 if (code < 0 || code >= 0x110000)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
425 index = 0;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
426 else {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
427 index = decomp_index1[(code>>DECOMP_SHIFT)];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
428 index = decomp_index2[(index<<DECOMP_SHIFT)+
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
429 (code&((1<<DECOMP_SHIFT)-1))];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
430 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
431
21112
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
432 /* high byte is number of hex bytes (usually one or two), low byte
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
433 is prefix code (from*/
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
434 count = decomp_data[index] >> 8;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
435
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
436 /* XXX: could allocate the PyString up front instead
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
437 (strlen(prefix) + 5 * count + 1 bytes) */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
438
39203
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
439 /* Based on how index is calculated above and decomp_data is generated
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
440 from Tools/unicode/makeunicodedata.py, it should not be possible
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
441 to overflow decomp_prefix. */
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
442 prefix_index = decomp_data[index] & 255;
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
443 assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix)));
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
444
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
445 /* copy prefix */
39203
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
446 i = strlen(decomp_prefix[prefix_index]);
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev
Neal Norwitz <nnorwitz@gmail.com>
parents: 38422
diff changeset
447 memcpy(decomp, decomp_prefix[prefix_index], i);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
448
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
449 while (count-- > 0) {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
450 if (i)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
451 decomp[i++] = ' ';
21112
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
452 assert((size_t)i < sizeof(decomp));
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
453 PyOS_snprintf(decomp + i, sizeof(decomp) - i, "%04X",
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
454 decomp_data[++index]);
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
455 i += strlen(decomp + i);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
456 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
457
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
458 decomp[i] = '\0';
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
459
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47451
diff changeset
460 return PyString_FromString(decomp);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
461 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
462
37780
b95b5c33cb3a No reason to export get_decomp_record, make static
Neal Norwitz <nnorwitz@gmail.com>
parents: 37664
diff changeset
463 static void
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
464 get_decomp_record(PyObject *self, Py_UCS4 code, int *index, int *prefix, int *count)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
465 {
28100
54b35da9879e Fix SF bug #694816, remove comparison of unsigned value < 0
Neal Norwitz <nnorwitz@gmail.com>
parents: 26571
diff changeset
466 if (code >= 0x110000) {
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
467 *index = 0;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
468 } else if (self && get_old_record(self, code)->category_changed==0) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
469 /* unassigned in old version */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
470 *index = 0;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
471 }
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
472 else {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
473 *index = decomp_index1[(code>>DECOMP_SHIFT)];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
474 *index = decomp_index2[(*index<<DECOMP_SHIFT)+
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
475 (code&((1<<DECOMP_SHIFT)-1))];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
476 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
477
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
478 /* high byte is number of hex bytes (usually one or two), low byte
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
479 is prefix code (from*/
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
480 *count = decomp_data[*index] >> 8;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
481 *prefix = decomp_data[*index] & 255;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
482
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
483 (*index)++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
484 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
485
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
486 #define SBase 0xAC00
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
487 #define LBase 0x1100
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
488 #define VBase 0x1161
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
489 #define TBase 0x11A7
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
490 #define LCount 19
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
491 #define VCount 21
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
492 #define TCount 28
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
493 #define NCount (VCount*TCount)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
494 #define SCount (LCount*NCount)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
495
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
496 static PyObject*
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
497 nfd_nfkd(PyObject *self, PyObject *input, int k)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
498 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
499 PyObject *result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
500 Py_UNICODE *i, *end, *o;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
501 /* Longest decomposition in Unicode 3.2: U+FDFA */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
502 Py_UNICODE stack[20];
37664
d13ded9165ec Support NFD of very long strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 37556
diff changeset
503 Py_ssize_t space, isize;
d13ded9165ec Support NFD of very long strings.
Martin v. Löwis <martin@v.loewis.de>
parents: 37556
diff changeset
504 int index, prefix, count, stackptr;
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
505 unsigned char prev, cur;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
506
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
507 stackptr = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
508 isize = PyUnicode_GET_SIZE(input);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
509 /* Overallocate atmost 10 characters. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
510 space = (isize > 10 ? 10 : isize) + isize;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
511 result = PyUnicode_FromUnicode(NULL, space);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
512 if (!result)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
513 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
514 i = PyUnicode_AS_UNICODE(input);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
515 end = i + isize;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
516 o = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
517
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
518 while (i < end) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
519 stack[stackptr++] = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
520 while(stackptr) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
521 Py_UNICODE code = stack[--stackptr];
30867
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 28100
diff changeset
522 /* Hangul Decomposition adds three characters in
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 28100
diff changeset
523 a single step, so we need atleast that much room. */
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 28100
diff changeset
524 if (space < 3) {
47527
52294af4b663 This reverts r63675 based on the discussion in this thread:
Gregory P. Smith <greg@mad-scientist.com>
parents: 47451
diff changeset
525 Py_ssize_t newsize = PyString_GET_SIZE(result) + 10;
30867
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 28100
diff changeset
526 space += 10;
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 28100
diff changeset
527 if (PyUnicode_Resize(&result, newsize) == -1)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
528 return NULL;
30867
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 28100
diff changeset
529 o = PyUnicode_AS_UNICODE(result) + newsize - space;
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
530 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
531 /* Hangul Decomposition. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
532 if (SBase <= code && code < (SBase+SCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
533 int SIndex = code - SBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
534 int L = LBase + SIndex / NCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
535 int V = VBase + (SIndex % NCount) / TCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
536 int T = TBase + SIndex % TCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
537 *o++ = L;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
538 *o++ = V;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
539 space -= 2;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
540 if (T != TBase) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
541 *o++ = T;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
542 space --;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
543 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
544 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
545 }
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
546 /* normalization changes */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
547 if (self) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
548 Py_UCS4 value = ((PreviousDBVersion*)self)->normalization(code);
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
549 if (value != 0) {
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
550 stack[stackptr++] = value;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
551 continue;
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
552 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
553 }
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
554
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
555 /* Other decompositions. */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
556 get_decomp_record(self, code, &index, &prefix, &count);
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
557
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
558 /* Copy character if it is not decomposable, or has a
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
559 compatibility decomposition, but we do NFD. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
560 if (!count || (prefix && !k)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
561 *o++ = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
562 space--;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
563 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
564 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
565 /* Copy decomposition onto the stack, in reverse
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
566 order. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
567 while(count) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
568 code = decomp_data[index + (--count)];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
569 stack[stackptr++] = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
570 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
571 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
572 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
573
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
574 /* Drop overallocation. Cannot fail. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
575 PyUnicode_Resize(&result, PyUnicode_GET_SIZE(result) - space);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
576
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
577 /* Sort canonically. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
578 i = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
579 prev = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
580 end = i + PyUnicode_GET_SIZE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
581 for (i++; i < end; i++) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
582 cur = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
583 if (prev == 0 || cur == 0 || prev <= cur) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
584 prev = cur;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
585 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
586 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
587 /* Non-canonical order. Need to switch *i with previous. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
588 o = i - 1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
589 while (1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
590 Py_UNICODE tmp = o[1];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
591 o[1] = o[0];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
592 o[0] = tmp;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
593 o--;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
594 if (o < PyUnicode_AS_UNICODE(result))
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
595 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
596 prev = _getrecord_ex(*o)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
597 if (prev == 0 || prev <= cur)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
598 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
599 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
600 prev = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
601 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
602 return result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
603 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
604
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
605 static int
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
606 find_nfc_index(PyObject *self, struct reindex* nfc, Py_UNICODE code)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
607 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
608 int index;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
609 for (index = 0; nfc[index].start; index++) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
610 int start = nfc[index].start;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
611 if (code < start)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
612 return -1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
613 if (code <= start + nfc[index].count) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
614 int delta = code - start;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
615 return nfc[index].index + delta;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
616 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
617 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
618 return -1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
619 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
620
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
621 static PyObject*
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
622 nfc_nfkc(PyObject *self, PyObject *input, int k)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
623 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
624 PyObject *result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
625 Py_UNICODE *i, *i1, *o, *end;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
626 int f,l,index,index1,comb;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
627 Py_UNICODE code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
628 Py_UNICODE *skipped[20];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
629 int cskipped = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
630
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
631 result = nfd_nfkd(self, input, k);
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
632 if (!result)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
633 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
634
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
635 /* We are going to modify result in-place.
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
636 If nfd_nfkd is changed to sometimes return the input,
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
637 this code needs to be reviewed. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
638 assert(result != input);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
639
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
640 i = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
641 end = i + PyUnicode_GET_SIZE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
642 o = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
643
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
644 again:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
645 while (i < end) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
646 for (index = 0; index < cskipped; index++) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
647 if (skipped[index] == i) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
648 /* *i character is skipped.
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
649 Remove from list. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
650 skipped[index] = skipped[cskipped-1];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
651 cskipped--;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
652 i++;
26571
cb8cfadf75a2 Remove C++ comment.
Martin v. Löwis <martin@v.loewis.de>
parents: 26397
diff changeset
653 goto again; /* continue while */
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
654 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
655 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
656 /* Hangul Composition. We don't need to check for <LV,T>
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
657 pairs, since we always have decomposed data. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
658 if (LBase <= *i && *i < (LBase+LCount) &&
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
659 i + 1 < end &&
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
660 VBase <= i[1] && i[1] <= (VBase+VCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
661 int LIndex, VIndex;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
662 LIndex = i[0] - LBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
663 VIndex = i[1] - VBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
664 code = SBase + (LIndex*VCount+VIndex)*TCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
665 i+=2;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
666 if (i < end &&
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
667 TBase <= *i && *i <= (TBase+TCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
668 code += *i-TBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
669 i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
670 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
671 *o++ = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
672 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
673 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
674
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
675 f = find_nfc_index(self, nfc_first, *i);
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
676 if (f == -1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
677 *o++ = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
678 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
679 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
680 /* Find next unblocked character. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
681 i1 = i+1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
682 comb = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
683 while (i1 < end) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
684 int comb1 = _getrecord_ex(*i1)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
685 if (comb1 && comb == comb1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
686 /* Character is blocked. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
687 i1++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
688 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
689 }
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
690 l = find_nfc_index(self, nfc_last, *i1);
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
691 /* *i1 cannot be combined with *i. If *i1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
692 is a starter, we don't need to look further.
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
693 Otherwise, record the combining class. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
694 if (l == -1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
695 not_combinable:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
696 if (comb1 == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
697 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
698 comb = comb1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
699 i1++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
700 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
701 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
702 index = f*TOTAL_LAST + l;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
703 index1 = comp_index[index >> COMP_SHIFT];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
704 code = comp_data[(index1<<COMP_SHIFT)+
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
705 (index&((1<<COMP_SHIFT)-1))];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
706 if (code == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
707 goto not_combinable;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
708
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
709 /* Replace the original character. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
710 *i = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
711 /* Mark the second character unused. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
712 skipped[cskipped++] = i1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
713 i1++;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
714 f = find_nfc_index(self, nfc_first, *i);
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
715 if (f == -1)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
716 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
717 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
718 *o++ = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
719 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
720 if (o != end)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
721 PyUnicode_Resize(&result, o - PyUnicode_AS_UNICODE(result));
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
722 return result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
723 }
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
724
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
725 /* Return 1 if the input is certainly normalized, 0 if it might not be. */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
726 static int
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
727 is_normalized(PyObject *self, PyObject *input, int nfc, int k)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
728 {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
729 Py_UNICODE *i, *end;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
730 unsigned char prev_combining = 0, quickcheck_mask;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
731
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
732 /* An older version of the database is requested, quickchecks must be
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
733 disabled. */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
734 if (self != NULL)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
735 return 0;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
736
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
737 /* The two quickcheck bits at this shift mean 0=Yes, 1=Maybe, 2=No,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
738 as described in http://unicode.org/reports/tr15/#Annex8. */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
739 quickcheck_mask = 3 << ((nfc ? 4 : 0) + (k ? 2 : 0));
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
740
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
741 i = PyUnicode_AS_UNICODE(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
742 end = i + PyUnicode_GET_SIZE(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
743 while (i < end) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
744 const _PyUnicode_DatabaseRecord *record = _getrecord_ex(*i++);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
745 unsigned char combining = record->combining;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
746 unsigned char quickcheck = record->normalization_quick_check;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
747
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
748 if (quickcheck & quickcheck_mask)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
749 return 0; /* this string might need normalization */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
750 if (combining && prev_combining > combining)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
751 return 0; /* non-canonical sort order, not normalized */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
752 prev_combining = combining;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
753 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
754 return 1; /* certainly normalized */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
755 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
756
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
757 PyDoc_STRVAR(unicodedata_normalize__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
758 "normalize(form, unistr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
759 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
760 Return the normal form 'form' for the Unicode string unistr. Valid\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
761 values for form are 'NFC', 'NFKC', 'NFD', and 'NFKD'.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
762
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
763 static PyObject*
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
764 unicodedata_normalize(PyObject *self, PyObject *args)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
765 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
766 char *form;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
767 PyObject *input;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
768
32680
d3d426483d71 Fix typo.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31930
diff changeset
769 if(!PyArg_ParseTuple(args, "sO!:normalize",
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
770 &form, &PyUnicode_Type, &input))
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
771 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
772
31930
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
773 if (PyUnicode_GetSize(input) == 0) {
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
774 /* Special case empty input strings, since resizing
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
775 them later would cause internal errors. */
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
776 Py_INCREF(input);
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
777 return input;
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
778 }
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361.
Martin v. Löwis <martin@v.loewis.de>
parents: 30867
diff changeset
779
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
780 if (strcmp(form, "NFC") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
781 if (is_normalized(self, input, 1, 0)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
782 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
783 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
784 }
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
785 return nfc_nfkc(self, input, 0);
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
786 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
787 if (strcmp(form, "NFKC") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
788 if (is_normalized(self, input, 1, 1)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
789 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
790 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
791 }
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
792 return nfc_nfkc(self, input, 1);
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
793 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
794 if (strcmp(form, "NFD") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
795 if (is_normalized(self, input, 0, 0)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
796 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
797 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
798 }
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
799 return nfd_nfkd(self, input, 0);
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
800 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
801 if (strcmp(form, "NFKD") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
802 if (is_normalized(self, input, 0, 1)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
803 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
804 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
805 }
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
806 return nfd_nfkd(self, input, 1);
53553
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the
Antoine Pitrou <solipsis@pitrou.net>
parents: 49124
diff changeset
807 }
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
808 PyErr_SetString(PyExc_ValueError, "invalid normalization form");
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
809 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
810 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
811
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
812 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
813 /* unicode character name tables */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
814
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
815 /* data file generated by Tools/unicode/makeunicodedata.py */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
816 #include "unicodename_db.h"
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
817
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
818 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
819 /* database code (cut and pasted from the unidb package) */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
820
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
821 static unsigned long
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
822 _gethash(const char *s, int len, int scale)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
823 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
824 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
825 unsigned long h = 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
826 unsigned long ix;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
827 for (i = 0; i < len; i++) {
37556
9bda5ba41677 Get rid of warnings about using chars as subscripts
Neal Norwitz <nnorwitz@gmail.com>
parents: 36939
diff changeset
828 h = (h * scale) + (unsigned char) toupper(Py_CHARMASK(s[i]));
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
829 ix = h & 0xff000000;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
830 if (ix)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
831 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
832 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
833 return h;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
834 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
835
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
836 static char *hangul_syllables[][3] = {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
837 { "G", "A", "" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
838 { "GG", "AE", "G" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
839 { "N", "YA", "GG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
840 { "D", "YAE", "GS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
841 { "DD", "EO", "N", },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
842 { "R", "E", "NJ" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
843 { "M", "YEO", "NH" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
844 { "B", "YE", "D" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
845 { "BB", "O", "L" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
846 { "S", "WA", "LG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
847 { "SS", "WAE", "LM" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
848 { "", "OE", "LB" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
849 { "J", "YO", "LS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
850 { "JJ", "U", "LT" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
851 { "C", "WEO", "LP" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
852 { "K", "WE", "LH" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
853 { "T", "WI", "M" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
854 { "P", "YU", "B" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
855 { "H", "EU", "BS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
856 { 0, "YI", "S" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
857 { 0, "I", "SS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
858 { 0, 0, "NG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
859 { 0, 0, "J" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
860 { 0, 0, "C" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
861 { 0, 0, "K" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
862 { 0, 0, "T" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
863 { 0, 0, "P" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
864 { 0, 0, "H" }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
865 };
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
866
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
867 static int
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
868 is_unified_ideograph(Py_UCS4 code)
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
869 {
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
870 return (
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
871 (0x3400 <= code && code <= 0x4DB5) || /* CJK Ideograph Extension A */
36939
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
872 (0x4E00 <= code && code <= 0x9FBB) || /* CJK Ideograph */
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
873 (0x20000 <= code && code <= 0x2A6D6));/* CJK Ideograph Extension B */
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
874 }
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
875
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
876 static int
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
877 _getucname(PyObject *self, Py_UCS4 code, char* buffer, int buflen)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
878 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
879 int offset;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
880 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
881 int word;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
882 unsigned char* w;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
883
36939
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
884 if (code >= 0x110000)
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
885 return 0;
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
886
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
887 if (self) {
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
888 const change_record *old = get_old_record(self, code);
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
889 if (old->category_changed == 0) {
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
890 /* unassigned */
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
891 return 0;
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
892 }
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
893 }
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36928
diff changeset
894
26374
135bb11215a0 Fix off-by-one error.
Martin v. Löwis <martin@v.loewis.de>
parents: 26370
diff changeset
895 if (SBase <= code && code < SBase+SCount) {
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
896 /* Hangul syllable. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
897 int SIndex = code - SBase;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
898 int L = SIndex / NCount;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
899 int V = (SIndex % NCount) / TCount;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
900 int T = SIndex % TCount;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
901
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
902 if (buflen < 27)
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
903 /* Worst case: HANGUL SYLLABLE <10chars>. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
904 return 0;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
905 strcpy(buffer, "HANGUL SYLLABLE ");
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
906 buffer += 16;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
907 strcpy(buffer, hangul_syllables[L][0]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
908 buffer += strlen(hangul_syllables[L][0]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
909 strcpy(buffer, hangul_syllables[V][1]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
910 buffer += strlen(hangul_syllables[V][1]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
911 strcpy(buffer, hangul_syllables[T][2]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
912 buffer += strlen(hangul_syllables[T][2]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
913 *buffer = '\0';
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
914 return 1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
915 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
916
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
917 if (is_unified_ideograph(code)) {
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
918 if (buflen < 28)
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
919 /* Worst case: CJK UNIFIED IDEOGRAPH-20000 */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
920 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
921 sprintf(buffer, "CJK UNIFIED IDEOGRAPH-%X", code);
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
922 return 1;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
923 }
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
924
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
925 /* get offset into phrasebook */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
926 offset = phrasebook_offset1[(code>>phrasebook_shift)];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
927 offset = phrasebook_offset2[(offset<<phrasebook_shift) +
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
928 (code&((1<<phrasebook_shift)-1))];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
929 if (!offset)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
930 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
931
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
932 i = 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
933
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
934 for (;;) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
935 /* get word index */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
936 word = phrasebook[offset] - phrasebook_short;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
937 if (word >= 0) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
938 word = (word << 8) + phrasebook[offset+1];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
939 offset += 2;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
940 } else
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
941 word = phrasebook[offset++];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
942 if (i) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
943 if (i > buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
944 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
945 buffer[i++] = ' ';
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
946 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
947 /* copy word string from lexicon. the last character in the
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
948 word has bit 7 set. the last word in a string ends with
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
949 0x80 */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
950 w = lexicon + lexicon_offset[word];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
951 while (*w < 128) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
952 if (i >= buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
953 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
954 buffer[i++] = *w++;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
955 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
956 if (i >= buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
957 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
958 buffer[i++] = *w & 127;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
959 if (*w == 128)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
960 break; /* end of word */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
961 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
962
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
963 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
964 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
965
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
966 static int
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
967 _cmpname(PyObject *self, int code, const char* name, int namelen)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
968 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
969 /* check if code corresponds to the given name */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
970 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
971 char buffer[NAME_MAXLEN];
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
972 if (!_getucname(self, code, buffer, sizeof(buffer)))
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
973 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
974 for (i = 0; i < namelen; i++) {
37556
9bda5ba41677 Get rid of warnings about using chars as subscripts
Neal Norwitz <nnorwitz@gmail.com>
parents: 36939
diff changeset
975 if (toupper(Py_CHARMASK(name[i])) != buffer[i])
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
976 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
977 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
978 return buffer[namelen] == '\0';
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
979 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
980
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
981 static void
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
982 find_syllable(const char *str, int *len, int *pos, int count, int column)
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
983 {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
984 int i, len1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
985 *len = -1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
986 for (i = 0; i < count; i++) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
987 char *s = hangul_syllables[i][column];
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
988 len1 = strlen(s);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
989 if (len1 <= *len)
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
990 continue;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
991 if (strncmp(str, s, len1) == 0) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
992 *len = len1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
993 *pos = i;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
994 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
995 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
996 if (*len == -1) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
997 *len = 0;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
998 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
999 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1000
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1001 static int
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1002 _getcode(PyObject* self, const char* name, int namelen, Py_UCS4* code)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1003 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1004 unsigned int h, v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1005 unsigned int mask = code_size-1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1006 unsigned int i, incr;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1007
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1008 /* Check for hangul syllables. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1009 if (strncmp(name, "HANGUL SYLLABLE ", 16) == 0) {
36778
cd7d2919178c Remove gcc (4.0.x) warning about uninitialized value by explicitly setting
Thomas Wouters <thomas@python.org>
parents: 35587
diff changeset
1010 int len, L = -1, V = -1, T = -1;
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1011 const char *pos = name + 16;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1012 find_syllable(pos, &len, &L, LCount, 0);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1013 pos += len;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1014 find_syllable(pos, &len, &V, VCount, 1);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1015 pos += len;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1016 find_syllable(pos, &len, &T, TCount, 2);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1017 pos += len;
35587
02a8c0643f5b Patch #1213831: Fix typo in unicodedata._getcode.
Martin v. Löwis <martin@v.loewis.de>
parents: 35016
diff changeset
1018 if (L != -1 && V != -1 && T != -1 && pos-name == namelen) {
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1019 *code = SBase + (L*VCount+V)*TCount + T;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1020 return 1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1021 }
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1022 /* Otherwise, it's an illegal syllable name. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1023 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1024 }
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1025
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1026 /* Check for unified ideographs. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1027 if (strncmp(name, "CJK UNIFIED IDEOGRAPH-", 22) == 0) {
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1028 /* Four or five hexdigits must follow. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1029 v = 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1030 name += 22;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1031 namelen -= 22;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1032 if (namelen != 4 && namelen != 5)
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1033 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1034 while (namelen--) {
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1035 v *= 16;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1036 if (*name >= '0' && *name <= '9')
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1037 v += *name - '0';
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1038 else if (*name >= 'A' && *name <= 'F')
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1039 v += *name - 'A' + 10;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1040 else
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1041 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1042 name++;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1043 }
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
1044 if (!is_unified_ideograph(v))
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
1045 return 0;
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1046 *code = v;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
1047 return 1;
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1048 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1049
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1050 /* the following is the same as python's dictionary lookup, with
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1051 only minor changes. see the makeunicodedata script for more
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1052 details */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1053
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
1054 h = (unsigned int) _gethash(name, namelen, code_magic);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1055 i = (~h) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1056 v = code_hash[i];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1057 if (!v)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1058 return 0;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1059 if (_cmpname(self, v, name, namelen)) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1060 *code = v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1061 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1062 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1063 incr = (h ^ (h >> 3)) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1064 if (!incr)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1065 incr = mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1066 for (;;) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1067 i = (i + incr) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1068 v = code_hash[i];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1069 if (!v)
16662
89c1ccef5ce8 stupid typo (for some reason, this only caused problems on OpenVMS).
Fredrik Lundh <fredrik@pythonware.com>
parents: 16200
diff changeset
1070 return 0;
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1071 if (_cmpname(self, v, name, namelen)) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1072 *code = v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1073 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1074 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1075 incr = incr << 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1076 if (incr > mask)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1077 incr = incr ^ code_poly;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1078 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1079 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1080
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1081 static const _PyUnicode_Name_CAPI hashAPI =
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1082 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1083 sizeof(_PyUnicode_Name_CAPI),
23810
01e4b698ff7e _Py prefix is verboten for static entry points
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22781
diff changeset
1084 _getucname,
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
1085 _getcode
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1086 };
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1087
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1088 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1089 /* Python bindings */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1090
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1091 PyDoc_STRVAR(unicodedata_name__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1092 "name(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1093 Returns the name assigned to the Unicode character unichr as a\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1094 string. If no name is defined, default is returned, or, if not\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1095 given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1096
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1097 static PyObject *
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1098 unicodedata_name(PyObject* self, PyObject* args)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1099 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1100 char name[NAME_MAXLEN];
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
1101 Py_UCS4 c;
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1102
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1103 PyUnicodeObject* v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1104 PyObject* defobj = NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1105 if (!PyArg_ParseTuple(args, "O!|O:name", &PyUnicode_Type, &v, &defobj))
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1106 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1107
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
1108 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
1109 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
1110 return NULL;
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1111
47451
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of
Walter Dörwald <walter@livinglogic.de>
parents: 47294
diff changeset
1112 if (!_getucname(self, c, name, sizeof(name))) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1113 if (defobj == NULL) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1114 PyErr_SetString(PyExc_ValueError, "no such name");
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1115 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1116 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1117 else {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1118 Py_INCREF(defobj);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1119 return defobj;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1120 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1121 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1122
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1123 return Py_BuildValue("s", name);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1124 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1125
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1126 PyDoc_STRVAR(unicodedata_lookup__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1127 "lookup(name)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1128 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1129 Look up character by name. If a character with the\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1130 given name is found, return the corresponding Unicode\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1131 character. If not found, KeyError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1132
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1133 static PyObject *
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1134 unicodedata_lookup(PyObject* self, PyObject* args)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1135 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1136 Py_UCS4 code;
42387
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1137 Py_UNICODE str[2];
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1138
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1139 char* name;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1140 int namelen;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1141 if (!PyArg_ParseTuple(args, "s#:lookup", &name, &namelen))
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1142 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1143
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1144 if (!_getcode(self, name, namelen, &code)) {
42387
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1145 PyErr_Format(PyExc_KeyError, "undefined character name '%s'",
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1146 name);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1147 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1148 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1149
42387
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1150 #ifndef Py_UNICODE_WIDE
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1151 if (code >= 0x10000) {
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1152 str[0] = 0xd800 + ((code - 0x10000) >> 10);
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1153 str[1] = 0xdc00 + ((code - 0x10000) & 0x3ff);
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1154 return PyUnicode_FromUnicode(str, 2);
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1155 }
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1156 #endif
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1157 str[0] = (Py_UNICODE) code;
42387
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot
Martin v. Löwis <martin@v.loewis.de>
parents: 42333
diff changeset
1158 return PyUnicode_FromUnicode(str, 1);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1159 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1160
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1161 /* XXX Add doc strings. */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1162
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1163 static PyMethodDef unicodedata_functions[] = {
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1164 {"decimal", unicodedata_decimal, METH_VARARGS, unicodedata_decimal__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1165 {"digit", unicodedata_digit, METH_VARARGS, unicodedata_digit__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1166 {"numeric", unicodedata_numeric, METH_VARARGS, unicodedata_numeric__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1167 {"category", unicodedata_category, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1168 unicodedata_category__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1169 {"bidirectional", unicodedata_bidirectional, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1170 unicodedata_bidirectional__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1171 {"combining", unicodedata_combining, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1172 unicodedata_combining__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1173 {"mirrored", unicodedata_mirrored, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1174 unicodedata_mirrored__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1175 {"east_asian_width", unicodedata_east_asian_width, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1176 unicodedata_east_asian_width__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1177 {"decomposition", unicodedata_decomposition, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1178 unicodedata_decomposition__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1179 {"name", unicodedata_name, METH_VARARGS, unicodedata_name__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1180 {"lookup", unicodedata_lookup, METH_VARARGS, unicodedata_lookup__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1181 {"normalize", unicodedata_normalize, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1182 unicodedata_normalize__doc__},
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1183 {NULL, NULL} /* sentinel */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1184 };
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1185
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1186 static PyTypeObject UCD_Type = {
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1187 /* The ob_type field must be initialized in the module init function
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1188 * to be portable to Windows without using C++. */
42333
6620a510e9a9 PEP 3123: Provide forward compatibility with Python 3.0, while keeping
Martin v. Löwis <martin@v.loewis.de>
parents: 40319
diff changeset
1189 PyVarObject_HEAD_INIT(NULL, 0)
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1190 "unicodedata.UCD", /*tp_name*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1191 sizeof(PreviousDBVersion), /*tp_basicsize*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1192 0, /*tp_itemsize*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1193 /* methods */
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1194 (destructor)PyObject_Del, /*tp_dealloc*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1195 0, /*tp_print*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1196 0, /*tp_getattr*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1197 0, /*tp_setattr*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1198 0, /*tp_compare*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1199 0, /*tp_repr*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1200 0, /*tp_as_number*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1201 0, /*tp_as_sequence*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1202 0, /*tp_as_mapping*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1203 0, /*tp_hash*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1204 0, /*tp_call*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1205 0, /*tp_str*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1206 PyObject_GenericGetAttr,/*tp_getattro*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1207 0, /*tp_setattro*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1208 0, /*tp_as_buffer*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1209 Py_TPFLAGS_DEFAULT, /*tp_flags*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1210 0, /*tp_doc*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1211 0, /*tp_traverse*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1212 0, /*tp_clear*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1213 0, /*tp_richcompare*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1214 0, /*tp_weaklistoffset*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1215 0, /*tp_iter*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1216 0, /*tp_iternext*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1217 unicodedata_functions, /*tp_methods*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1218 DB_members, /*tp_members*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1219 0, /*tp_getset*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1220 0, /*tp_base*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1221 0, /*tp_dict*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1222 0, /*tp_descr_get*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1223 0, /*tp_descr_set*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1224 0, /*tp_dictoffset*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1225 0, /*tp_init*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1226 0, /*tp_alloc*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1227 0, /*tp_new*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1228 0, /*tp_free*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1229 0, /*tp_is_gc*/
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1230 };
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1231
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1232 PyDoc_STRVAR(unicodedata_docstring,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1233 "This module provides access to the Unicode Character Database which\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1234 defines character properties for all Unicode characters. The data in\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1235 this database is based on the UnicodeData.txt file version\n\
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
1236 5.1.0 which is publically available from ftp://ftp.unicode.org/.\n\
34864
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1237 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32951
diff changeset
1238 The module uses the same names and symbols as defined by the\n\
49124
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
1239 UnicodeData File Format 5.1.0 (see\n\
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 47527
diff changeset
1240 http://www.unicode.org/Public/5.1.0/ucd/UCD.html).");
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1241
24363
979ff5d0ae07 Replace DL_IMPORT with PyMODINIT_FUNC and remove "/export:init..." link
Mark Hammond <mhammond@skippinet.com.au>
parents: 23832
diff changeset
1242 PyMODINIT_FUNC
13651
8a063423e0b7 Bunch of minor ANSIfications: 'void initfunc()' -> 'void initfunc(void)',
Thomas Wouters <thomas@python.org>
parents: 12196
diff changeset
1243 initunicodedata(void)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1244 {
22781
ec67fce3fa3c Remove direct manipulation of the module dict.
Fred Drake <fdrake@acm.org>
parents: 22114
diff changeset
1245 PyObject *m, *v;
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1246
44350
26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.
Christian Heimes <christian@cheimes.de>
parents: 42387
diff changeset
1247 Py_TYPE(&UCD_Type) = &PyType_Type;
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1248
16977
0b4c57a25977 Be a bit more strict in setting up the export of the C API for this
Fred Drake <fdrake@acm.org>
parents: 16666
diff changeset
1249 m = Py_InitModule3(
0b4c57a25977 Be a bit more strict in setting up the export of the C API for this
Fred Drake <fdrake@acm.org>
parents: 16666
diff changeset
1250 "unicodedata", unicodedata_functions, unicodedata_docstring);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1251 if (!m)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1252 return;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1253
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26380
diff changeset
1254 PyModule_AddStringConstant(m, "unidata_version", UNIDATA_VERSION);
36928
11f8b5b8ce20 Fix refcounting bug.
Martin v. Löwis <martin@v.loewis.de>
parents: 36927
diff changeset
1255 Py_INCREF(&UCD_Type);
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1256 PyModule_AddObject(m, "UCD", (PyObject*)&UCD_Type);
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26380
diff changeset
1257
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1258 /* Previous versions */
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1259 v = new_previous_version("3.2.0", get_change_3_2_0, normalization_3_2_0);
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1260 if (v != NULL)
36927
3266a020a0da Avoid forward-declaring the methods array.
Martin v. Löwis <martin@v.loewis.de>
parents: 36924
diff changeset
1261 PyModule_AddObject(m, "ucd_3_2_0", v);
36924
48e0a5e39403 Update Unicode database to Unicode 4.1.
Martin v. Löwis <martin@v.loewis.de>
parents: 36778
diff changeset
1262
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1263 /* Export C API */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
1264 v = PyCObject_FromVoidPtr((void *) &hashAPI, NULL);
22781
ec67fce3fa3c Remove direct manipulation of the module dict.
Fred Drake <fdrake@acm.org>
parents: 22114
diff changeset
1265 if (v != NULL)
ec67fce3fa3c Remove direct manipulation of the module dict.
Fred Drake <fdrake@acm.org>
parents: 22114
diff changeset
1266 PyModule_AddObject(m, "ucnhash_CAPI", v);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1267 }
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1268
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1269 /*
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1270 Local variables:
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1271 c-basic-offset: 4
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
1272 indent-tabs-mode: nil
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1273 End:
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
1274 */