Mercurial > cpython-lru_cache-weakref

annotate Modules/unicodedata.c @ 27998:e27f58a50737 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
SF bug #690377: Dud footnote reference in distutils docs
author Raymond Hettinger <python@rcn.com>
date Fri, 21 Feb 2003 02:47:09 +0000
parents cb8cfadf75a2
children 54b35da9879e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 /* ------------------------------------------------------------------------
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
3 unicodedata -- Provides access to the Unicode 3.2 data base.
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
4
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
5 Data was extracted from the Unicode 3.2 UnicodeData.txt file.
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14818
diff changeset
7 Written by Marc-Andre Lemburg (mal@lemburg.com).
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14818
diff changeset
8 Modified for Python 2.0 by Fredrik Lundh (fredrik@pythonware.com)
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
9 Modified by Martin v. Löwis (martin@v.loewis.de)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
14835
96d31a143857 unicode database compression, step 2:
Fredrik Lundh <fredrik@pythonware.com>
parents: 14818
diff changeset
11 Copyright (c) Corporation for National Research Initiatives.
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
12
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
13 ------------------------------------------------------------------------ */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
14
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
15 #include "Python.h"
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
16 #include "ucnhash.h"
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
17
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
18 /* character properties */
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
19
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
20 typedef struct {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
21 const unsigned char category; /* index into
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
22 _PyUnicode_CategoryNames */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
23 const unsigned char combining; /* combining class value 0 - 255 */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
24 const unsigned char bidirectional; /* index into
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
25 _PyUnicode_BidirectionalNames */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
26 const unsigned char mirrored; /* true if mirrored in bidir mode */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
27 } _PyUnicode_DatabaseRecord;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
28
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
29 /* data file generated by Tools/unicode/makeunicodedata.py */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
30 #include "unicodedata_db.h"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
31
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
32 static const _PyUnicode_DatabaseRecord*
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
33 _getrecord_ex(Py_UCS4 code)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
34 {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
35 int index;
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 24363
diff changeset
36 if (code < 0 || code >= 0x110000)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
37 index = 0;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
38 else {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
39 index = index1[(code>>SHIFT)];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
40 index = index2[(index<<SHIFT)+(code&((1<<SHIFT)-1))];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
41 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
42
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
43 return &_PyUnicode_Database_Records[index];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
44 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
45
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
46 static const _PyUnicode_DatabaseRecord*
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
47 _getrecord(PyUnicodeObject* v)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
48 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
49 return _getrecord_ex(*PyUnicode_AS_UNICODE(v));
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
50 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
51
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
52 /* --- Module API --------------------------------------------------------- */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
53
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
55 unicodedata_decimal(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
56 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
57 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 PyObject *defobj = NULL;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 long rc;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
60
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
61 if (!PyArg_ParseTuple(args, "O!|O:decimal", &PyUnicode_Type, &v, &defobj))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
62 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
63 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
64 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
65 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
66 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
67 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
68 rc = Py_UNICODE_TODECIMAL(*PyUnicode_AS_UNICODE(v));
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
69 if (rc < 0) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
70 if (defobj == NULL) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
71 PyErr_SetString(PyExc_ValueError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
72 "not a decimal");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
73 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
74 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
75 else {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
76 Py_INCREF(defobj);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
77 return defobj;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
78 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
79 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
80 return PyInt_FromLong(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
81 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
82
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
83 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
84 unicodedata_digit(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
87 PyObject *defobj = NULL;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 long rc;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
89
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
90 if (!PyArg_ParseTuple(args, "O!|O:digit", &PyUnicode_Type, &v, &defobj))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
91 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
92 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
93 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
94 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
95 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
96 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
97 rc = Py_UNICODE_TODIGIT(*PyUnicode_AS_UNICODE(v));
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
98 if (rc < 0) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
99 if (defobj == NULL) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
100 PyErr_SetString(PyExc_ValueError, "not a digit");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
101 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
102 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
103 else {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
104 Py_INCREF(defobj);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
105 return defobj;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
106 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
107 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
108 return PyInt_FromLong(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
109 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
110
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
111 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
112 unicodedata_numeric(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
113 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
114 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
115 PyObject *defobj = NULL;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
116 double rc;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
117
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
118 if (!PyArg_ParseTuple(args, "O!|O:numeric", &PyUnicode_Type, &v, &defobj))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
119 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
120 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
123 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 rc = Py_UNICODE_TONUMERIC(*PyUnicode_AS_UNICODE(v));
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 if (rc < 0) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
127 if (defobj == NULL) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
128 PyErr_SetString(PyExc_ValueError, "not a numeric character");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
129 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
130 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 else {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 Py_INCREF(defobj);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 return defobj;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
134 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
136 return PyFloat_FromDouble(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
137 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
138
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
139 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
140 unicodedata_category(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
141 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
142 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
143 int index;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
144
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
145 if (!PyArg_ParseTuple(args, "O!:category",
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
146 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
147 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
149 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
150 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
151 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
152 }
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
153 index = (int) _getrecord(v)->category;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
154 return PyString_FromString(_PyUnicode_CategoryNames[index]);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
155 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
156
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
157 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
158 unicodedata_bidirectional(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
159 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
160 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
161 int index;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
162
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
163 if (!PyArg_ParseTuple(args, "O!:bidirectional",
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
164 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
165 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
166 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
167 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
169 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
170 }
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
171 index = (int) _getrecord(v)->bidirectional;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
172 return PyString_FromString(_PyUnicode_BidirectionalNames[index]);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
174
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
176 unicodedata_combining(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
177 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
178 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
179
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
180 if (!PyArg_ParseTuple(args, "O!:combining",
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
181 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
182 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
186 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
187 }
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
188 return PyInt_FromLong((int) _getrecord(v)->combining);
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
189 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
190
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
191 static PyObject *
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
192 unicodedata_mirrored(PyObject *self, PyObject *args)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
193 {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
194 PyUnicodeObject *v;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
195
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
196 if (!PyArg_ParseTuple(args, "O!:mirrored",
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
197 &PyUnicode_Type, &v))
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
198 return NULL;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
199 if (PyUnicode_GET_SIZE(v) != 1) {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
200 PyErr_SetString(PyExc_TypeError,
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
201 "need a single Unicode character as parameter");
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
202 return NULL;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
203 }
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
204 return PyInt_FromLong((int) _getrecord(v)->mirrored);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
206
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
207 static PyObject *
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
208 unicodedata_decomposition(PyObject *self, PyObject *args)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
210 PyUnicodeObject *v;
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
211 char decomp[256];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
212 int code, index, count, i;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
213
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
214 if (!PyArg_ParseTuple(args, "O!:decomposition",
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 &PyUnicode_Type, &v))
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
216 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
217 if (PyUnicode_GET_SIZE(v) != 1) {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
218 PyErr_SetString(PyExc_TypeError,
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
219 "need a single Unicode character as parameter");
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
220 return NULL;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
221 }
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
222
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
223 code = (int) *PyUnicode_AS_UNICODE(v);
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
224
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 24363
diff changeset
225 if (code < 0 || code >= 0x110000)
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
226 index = 0;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
227 else {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
228 index = decomp_index1[(code>>DECOMP_SHIFT)];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
229 index = decomp_index2[(index<<DECOMP_SHIFT)+
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
230 (code&((1<<DECOMP_SHIFT)-1))];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
231 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
232
21112
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
233 /* high byte is number of hex bytes (usually one or two), low byte
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
234 is prefix code (from*/
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
235 count = decomp_data[index] >> 8;
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
236
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
237 /* XXX: could allocate the PyString up front instead
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
238 (strlen(prefix) + 5 * count + 1 bytes) */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
239
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
240 /* copy prefix */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
241 i = strlen(decomp_prefix[decomp_data[index] & 255]);
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
242 memcpy(decomp, decomp_prefix[decomp_data[index] & 255], i);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
243
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
244 while (count-- > 0) {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
245 if (i)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
246 decomp[i++] = ' ';
21112
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
247 assert((size_t)i < sizeof(decomp));
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
248 PyOS_snprintf(decomp + i, sizeof(decomp) - i, "%04X",
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf.
Tim Peters <tim.peters@gmail.com>
parents: 18752
diff changeset
249 decomp_data[++index]);
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
250 i += strlen(decomp + i);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
251 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
252
16090
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
253 decomp[i] = '\0';
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
254
8ac40d48a581 compress unicode decomposition tables (this saves another 55k)
Fredrik Lundh <fredrik@pythonware.com>
parents: 14835
diff changeset
255 return PyString_FromString(decomp);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
256 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
257
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
258 void
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
259 get_decomp_record(Py_UCS4 code, int *index, int *prefix, int *count)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
260 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
261 if (code < 0 || code >= 0x110000) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
262 *index = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
263 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
264 else {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
265 *index = decomp_index1[(code>>DECOMP_SHIFT)];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
266 *index = decomp_index2[(*index<<DECOMP_SHIFT)+
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
267 (code&((1<<DECOMP_SHIFT)-1))];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
268 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
269
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
270 /* high byte is number of hex bytes (usually one or two), low byte
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
271 is prefix code (from*/
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
272 *count = decomp_data[*index] >> 8;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
273 *prefix = decomp_data[*index] & 255;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
274
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
275 (*index)++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
276 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
277
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
278 #define SBase 0xAC00
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
279 #define LBase 0x1100
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
280 #define VBase 0x1161
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
281 #define TBase 0x11A7
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
282 #define LCount 19
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
283 #define VCount 21
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
284 #define TCount 28
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
285 #define NCount (VCount*TCount)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
286 #define SCount (LCount*NCount)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
287
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
288 static PyObject*
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
289 nfd_nfkd(PyObject *input, int k)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
290 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
291 PyObject *result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
292 Py_UNICODE *i, *end, *o;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
293 /* Longest decomposition in Unicode 3.2: U+FDFA */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
294 Py_UNICODE stack[20];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
295 int space, stackptr, isize;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
296 int index, prefix, count;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
297 unsigned char prev, cur;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
298
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
299 stackptr = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
300 isize = PyUnicode_GET_SIZE(input);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
301 /* Overallocate atmost 10 characters. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
302 space = (isize > 10 ? 10 : isize) + isize;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
303 result = PyUnicode_FromUnicode(NULL, space);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
304 if (!result)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
305 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
306 i = PyUnicode_AS_UNICODE(input);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
307 end = i + isize;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
308 o = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
309
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
310 while (i < end) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
311 stack[stackptr++] = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
312 while(stackptr) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
313 Py_UNICODE code = stack[--stackptr];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
314 if (!space) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
315 space = PyString_GET_SIZE(result) + 10;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
316 if (PyUnicode_Resize(&result, space) == -1)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
317 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
318 o = PyUnicode_AS_UNICODE(result) + space - 10;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
319 space = 10;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
320 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
321 /* Hangul Decomposition. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
322 if (SBase <= code && code < (SBase+SCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
323 int SIndex = code - SBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
324 int L = LBase + SIndex / NCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
325 int V = VBase + (SIndex % NCount) / TCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
326 int T = TBase + SIndex % TCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
327 *o++ = L;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
328 *o++ = V;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
329 space -= 2;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
330 if (T != TBase) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
331 *o++ = T;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
332 space --;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
333 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
334 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
335 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
336 /* Other decompoistions. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
337 get_decomp_record(code, &index, &prefix, &count);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
338
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
339 /* Copy character if it is not decomposable, or has a
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
340 compatibility decomposition, but we do NFD. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
341 if (!count || (prefix && !k)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
342 *o++ = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
343 space--;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
344 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
345 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
346 /* Copy decomposition onto the stack, in reverse
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
347 order. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
348 while(count) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
349 code = decomp_data[index + (--count)];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
350 stack[stackptr++] = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
351 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
352 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
353 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
354
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
355 /* Drop overallocation. Cannot fail. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
356 PyUnicode_Resize(&result, PyUnicode_GET_SIZE(result) - space);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
357
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
358 /* Sort canonically. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
359 i = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
360 prev = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
361 end = i + PyUnicode_GET_SIZE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
362 for (i++; i < end; i++) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
363 cur = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
364 if (prev == 0 || cur == 0 || prev <= cur) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
365 prev = cur;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
366 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
367 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
368 /* Non-canonical order. Need to switch *i with previous. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
369 o = i - 1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
370 while (1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
371 Py_UNICODE tmp = o[1];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
372 o[1] = o[0];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
373 o[0] = tmp;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
374 o--;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
375 if (o < PyUnicode_AS_UNICODE(result))
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
376 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
377 prev = _getrecord_ex(*o)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
378 if (prev == 0 || prev <= cur)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
379 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
380 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
381 prev = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
382 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
383 return result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
384 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
385
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
386 static int
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
387 find_nfc_index(struct reindex* nfc, Py_UNICODE code)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
388 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
389 int index;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
390 for (index = 0; nfc[index].start; index++) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
391 int start = nfc[index].start;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
392 if (code < start)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
393 return -1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
394 if (code <= start + nfc[index].count) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
395 int delta = code - start;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
396 return nfc[index].index + delta;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
397 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
398 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
399 return -1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
400 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
401
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
402 static PyObject*
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
403 nfc_nfkc(PyObject *input, int k)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
404 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
405 PyObject *result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
406 Py_UNICODE *i, *i1, *o, *end;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
407 int f,l,index,index1,comb;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
408 Py_UNICODE code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
409 Py_UNICODE *skipped[20];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
410 int cskipped = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
411
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
412 result = nfd_nfkd(input, k);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
413 if (!result)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
414 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
415
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
416 /* We are going to modify result in-place.
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
417 If nfd_nfkd is changed to sometimes return the input,
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
418 this code needs to be reviewed. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
419 assert(result != input);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
420
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
421 i = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
422 end = i + PyUnicode_GET_SIZE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
423 o = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
424
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
425 again:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
426 while (i < end) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
427 for (index = 0; index < cskipped; index++) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
428 if (skipped[index] == i) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
429 /* *i character is skipped.
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
430 Remove from list. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
431 skipped[index] = skipped[cskipped-1];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
432 cskipped--;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
433 i++;
26571
cb8cfadf75a2 Remove C++ comment.
Martin v. Löwis <martin@v.loewis.de>
parents: 26397
diff changeset
434 goto again; /* continue while */
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
435 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
436 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
437 /* Hangul Composition. We don't need to check for <LV,T>
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
438 pairs, since we always have decomposed data. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
439 if (LBase <= *i && *i < (LBase+LCount) &&
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
440 i + 1 < end &&
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
441 VBase <= i[1] && i[1] <= (VBase+VCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
442 int LIndex, VIndex;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
443 LIndex = i[0] - LBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
444 VIndex = i[1] - VBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
445 code = SBase + (LIndex*VCount+VIndex)*TCount;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
446 i+=2;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
447 if (i < end &&
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
448 TBase <= *i && *i <= (TBase+TCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
449 code += *i-TBase;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
450 i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
451 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
452 *o++ = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
453 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
454 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
455
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
456 f = find_nfc_index(nfc_first, *i);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
457 if (f == -1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
458 *o++ = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
459 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
460 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
461 /* Find next unblocked character. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
462 i1 = i+1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
463 comb = 0;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
464 while (i1 < end) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
465 int comb1 = _getrecord_ex(*i1)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
466 if (comb1 && comb == comb1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
467 /* Character is blocked. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
468 i1++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
469 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
470 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
471 l = find_nfc_index(nfc_last, *i1);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
472 /* *i1 cannot be combined with *i. If *i1
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
473 is a starter, we don't need to look further.
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
474 Otherwise, record the combining class. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
475 if (l == -1) {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
476 not_combinable:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
477 if (comb1 == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
478 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
479 comb = comb1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
480 i1++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
481 continue;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
482 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
483 index = f*TOTAL_LAST + l;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
484 index1 = comp_index[index >> COMP_SHIFT];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
485 code = comp_data[(index1<<COMP_SHIFT)+
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
486 (index&((1<<COMP_SHIFT)-1))];
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
487 if (code == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
488 goto not_combinable;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
489
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
490 /* Replace the original character. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
491 *i = code;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
492 /* Mark the second character unused. */
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
493 skipped[cskipped++] = i1;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
494 i1++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
495 f = find_nfc_index(nfc_first, *i);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
496 if (f == -1)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
497 break;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
498 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
499 *o++ = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
500 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
501 if (o != end)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
502 PyUnicode_Resize(&result, o - PyUnicode_AS_UNICODE(result));
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
503 return result;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
504 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
505
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
506 static PyObject*
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
507 unicodedata_normalize(PyObject *self, PyObject *args)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
508 {
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
509 char *form;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
510 PyObject *input;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
511
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
512 if(!PyArg_ParseTuple(args, "sO!:normalized",
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
513 &form, &PyUnicode_Type, &input))
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
514 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
515
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
516 if (strcmp(form, "NFC") == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
517 return nfc_nfkc(input, 0);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
518 if (strcmp(form, "NFKC") == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
519 return nfc_nfkc(input, 1);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
520 if (strcmp(form, "NFD") == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
521 return nfd_nfkd(input, 0);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
522 if (strcmp(form, "NFKD") == 0)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
523 return nfd_nfkd(input, 1);
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
524 PyErr_SetString(PyExc_ValueError, "invalid normalization form");
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
525 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
526 }
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
527
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
528 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
529 /* unicode character name tables */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
530
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
531 /* data file generated by Tools/unicode/makeunicodedata.py */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
532 #include "unicodename_db.h"
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
533
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
534 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
535 /* database code (cut and pasted from the unidb package) */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
536
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
537 static unsigned long
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
538 _gethash(const char *s, int len, int scale)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
539 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
540 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
541 unsigned long h = 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
542 unsigned long ix;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
543 for (i = 0; i < len; i++) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
544 h = (h * scale) + (unsigned char) toupper(s[i]);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
545 ix = h & 0xff000000;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
546 if (ix)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
547 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
548 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
549 return h;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
550 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
551
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
552 static char *hangul_syllables[][3] = {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
553 { "G", "A", "" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
554 { "GG", "AE", "G" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
555 { "N", "YA", "GG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
556 { "D", "YAE", "GS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
557 { "DD", "EO", "N", },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
558 { "R", "E", "NJ" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
559 { "M", "YEO", "NH" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
560 { "B", "YE", "D" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
561 { "BB", "O", "L" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
562 { "S", "WA", "LG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
563 { "SS", "WAE", "LM" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
564 { "", "OE", "LB" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
565 { "J", "YO", "LS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
566 { "JJ", "U", "LT" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
567 { "C", "WEO", "LP" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
568 { "K", "WE", "LH" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
569 { "T", "WI", "M" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
570 { "P", "YU", "B" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
571 { "H", "EU", "BS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
572 { 0, "YI", "S" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
573 { 0, "I", "SS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
574 { 0, 0, "NG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
575 { 0, 0, "J" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
576 { 0, 0, "C" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
577 { 0, 0, "K" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
578 { 0, 0, "T" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
579 { 0, 0, "P" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
580 { 0, 0, "H" }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
581 };
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
582
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
583 static int
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
584 is_unified_ideograph(Py_UCS4 code)
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
585 {
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
586 return (
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
587 (0x3400 <= code && code <= 0x4DB5) || /* CJK Ideograph Extension A */
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
588 (0x4E00 <= code && code <= 0x9FA5) || /* CJK Ideograph */
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
589 (0x20000 <= code && code <= 0x2A6D6));/* CJK Ideograph Extension B */
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
590 }
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
591
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
592 static int
23810
01e4b698ff7e _Py prefix is verboten for static entry points
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22781
diff changeset
593 _getucname(Py_UCS4 code, char* buffer, int buflen)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
594 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
595 int offset;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
596 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
597 int word;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
598 unsigned char* w;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
599
26374
135bb11215a0 Fix off-by-one error.
Martin v. Löwis <martin@v.loewis.de>
parents: 26370
diff changeset
600 if (SBase <= code && code < SBase+SCount) {
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
601 /* Hangul syllable. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
602 int SIndex = code - SBase;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
603 int L = SIndex / NCount;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
604 int V = (SIndex % NCount) / TCount;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
605 int T = SIndex % TCount;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
606
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
607 if (buflen < 27)
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
608 /* Worst case: HANGUL SYLLABLE <10chars>. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
609 return 0;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
610 strcpy(buffer, "HANGUL SYLLABLE ");
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
611 buffer += 16;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
612 strcpy(buffer, hangul_syllables[L][0]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
613 buffer += strlen(hangul_syllables[L][0]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
614 strcpy(buffer, hangul_syllables[V][1]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
615 buffer += strlen(hangul_syllables[V][1]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
616 strcpy(buffer, hangul_syllables[T][2]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
617 buffer += strlen(hangul_syllables[T][2]);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
618 *buffer = '\0';
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
619 return 1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
620 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
621
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
622 if (is_unified_ideograph(code)) {
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
623 if (buflen < 28)
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
624 /* Worst case: CJK UNIFIED IDEOGRAPH-20000 */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
625 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
626 sprintf(buffer, "CJK UNIFIED IDEOGRAPH-%X", code);
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
627 return 1;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
628 }
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
629
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 24363
diff changeset
630 if (code >= 0x110000)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
631 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
632
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
633 /* get offset into phrasebook */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
634 offset = phrasebook_offset1[(code>>phrasebook_shift)];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
635 offset = phrasebook_offset2[(offset<<phrasebook_shift) +
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
636 (code&((1<<phrasebook_shift)-1))];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
637 if (!offset)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
638 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
639
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
640 i = 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
641
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
642 for (;;) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
643 /* get word index */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
644 word = phrasebook[offset] - phrasebook_short;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
645 if (word >= 0) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
646 word = (word << 8) + phrasebook[offset+1];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
647 offset += 2;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
648 } else
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
649 word = phrasebook[offset++];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
650 if (i) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
651 if (i > buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
652 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
653 buffer[i++] = ' ';
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
654 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
655 /* copy word string from lexicon. the last character in the
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
656 word has bit 7 set. the last word in a string ends with
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
657 0x80 */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
658 w = lexicon + lexicon_offset[word];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
659 while (*w < 128) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
660 if (i >= buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
661 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
662 buffer[i++] = *w++;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
663 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
664 if (i >= buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
665 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
666 buffer[i++] = *w & 127;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
667 if (*w == 128)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
668 break; /* end of word */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
669 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
670
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
671 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
672 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
673
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
674 static int
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
675 _cmpname(int code, const char* name, int namelen)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
676 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
677 /* check if code corresponds to the given name */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
678 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
679 char buffer[NAME_MAXLEN];
23810
01e4b698ff7e _Py prefix is verboten for static entry points
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22781
diff changeset
680 if (!_getucname(code, buffer, sizeof(buffer)))
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
681 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
682 for (i = 0; i < namelen; i++) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
683 if (toupper(name[i]) != buffer[i])
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
684 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
685 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
686 return buffer[namelen] == '\0';
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
687 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
688
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
689 static void
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
690 find_syllable(const char *str, int *len, int *pos, int count, int column)
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
691 {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
692 int i, len1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
693 *len = -1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
694 for (i = 0; i < count; i++) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
695 char *s = hangul_syllables[i][column];
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
696 len1 = strlen(s);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
697 if (len1 <= *len)
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
698 continue;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
699 if (strncmp(str, s, len1) == 0) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
700 *len = len1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
701 *pos = i;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
702 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
703 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
704 if (*len == -1) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
705 *len = 0;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
706 *pos = -1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
707 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
708 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
709
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
710 static int
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
711 _getcode(const char* name, int namelen, Py_UCS4* code)
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
712 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
713 unsigned int h, v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
714 unsigned int mask = code_size-1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
715 unsigned int i, incr;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
716
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
717 /* Check for hangul syllables. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
718 if (strncmp(name, "HANGUL SYLLABLE ", 16) == 0) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
719 int L, V, T, len;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
720 const char *pos = name + 16;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
721 find_syllable(pos, &len, &L, LCount, 0);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
722 pos += len;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
723 find_syllable(pos, &len, &V, VCount, 1);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
724 pos += len;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
725 find_syllable(pos, &len, &T, TCount, 2);
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
726 pos += len;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
727 if (V != -1 && V != -1 && T != -1 && pos-name == namelen) {
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
728 *code = SBase + (L*VCount+V)*TCount + T;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
729 return 1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
730 }
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
731 /* Otherwise, it's an illegal syllable name. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
732 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
733 }
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
734
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
735 /* Check for unified ideographs. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
736 if (strncmp(name, "CJK UNIFIED IDEOGRAPH-", 22) == 0) {
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
737 /* Four or five hexdigits must follow. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
738 v = 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
739 name += 22;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
740 namelen -= 22;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
741 if (namelen != 4 && namelen != 5)
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
742 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
743 while (namelen--) {
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
744 v *= 16;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
745 if (*name >= '0' && *name <= '9')
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
746 v += *name - '0';
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
747 else if (*name >= 'A' && *name <= 'F')
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
748 v += *name - 'A' + 10;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
749 else
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
750 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
751 name++;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
752 }
26380
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
753 if (!is_unified_ideograph(v))
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph.
Martin v. Löwis <martin@v.loewis.de>
parents: 26379
diff changeset
754 return 0;
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
755 *code = v;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
756 return 1;
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
757 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
758
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
759 /* the following is the same as python's dictionary lookup, with
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
760 only minor changes. see the makeunicodedata script for more
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
761 details */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
762
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
763 h = (unsigned int) _gethash(name, namelen, code_magic);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
764 i = (~h) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
765 v = code_hash[i];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
766 if (!v)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
767 return 0;
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
768 if (_cmpname(v, name, namelen)) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
769 *code = v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
770 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
771 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
772 incr = (h ^ (h >> 3)) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
773 if (!incr)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
774 incr = mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
775 for (;;) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
776 i = (i + incr) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
777 v = code_hash[i];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
778 if (!v)
16662
89c1ccef5ce8 stupid typo (for some reason, this only caused problems on OpenVMS).
Fredrik Lundh <fredrik@pythonware.com>
parents: 16200
diff changeset
779 return 0;
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
780 if (_cmpname(v, name, namelen)) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
781 *code = v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
782 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
783 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
784 incr = incr << 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
785 if (incr > mask)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
786 incr = incr ^ code_poly;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
787 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
788 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
789
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
790 static const _PyUnicode_Name_CAPI hashAPI =
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
791 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
792 sizeof(_PyUnicode_Name_CAPI),
23810
01e4b698ff7e _Py prefix is verboten for static entry points
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22781
diff changeset
793 _getucname,
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
794 _getcode
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
795 };
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
796
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
797 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
798 /* Python bindings */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
799
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
800 static PyObject *
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
801 unicodedata_name(PyObject* self, PyObject* args)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
802 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
803 char name[NAME_MAXLEN];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
804
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
805 PyUnicodeObject* v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
806 PyObject* defobj = NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
807 if (!PyArg_ParseTuple(args, "O!|O:name", &PyUnicode_Type, &v, &defobj))
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
808 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
809
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
810 if (PyUnicode_GET_SIZE(v) != 1) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
811 PyErr_SetString(PyExc_TypeError,
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
812 "need a single Unicode character as parameter");
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
813 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
814 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
815
23810
01e4b698ff7e _Py prefix is verboten for static entry points
Andrew MacIntyre <andymac@bullseye.apana.org.au>
parents: 22781
diff changeset
816 if (!_getucname((Py_UCS4) *PyUnicode_AS_UNICODE(v),
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
817 name, sizeof(name))) {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
818 if (defobj == NULL) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
819 PyErr_SetString(PyExc_ValueError, "no such name");
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
820 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
821 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
822 else {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
823 Py_INCREF(defobj);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
824 return defobj;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
825 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
826 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
827
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
828 return Py_BuildValue("s", name);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
829 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
830
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
831 static PyObject *
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
832 unicodedata_lookup(PyObject* self, PyObject* args)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
833 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
834 Py_UCS4 code;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
835 Py_UNICODE str[1];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
836
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
837 char* name;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
838 int namelen;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
839 if (!PyArg_ParseTuple(args, "s#:lookup", &name, &namelen))
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
840 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
841
16666
e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS
Fredrik Lundh <fredrik@pythonware.com>
parents: 16662
diff changeset
842 if (!_getcode(name, namelen, &code)) {
26376
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
843 char fmt[] = "undefined character name '%s'";
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
844 char *buf = PyMem_MALLOC(sizeof(fmt) + namelen);
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
845 sprintf(buf, fmt, name);
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
846 PyErr_SetString(PyExc_KeyError, buf);
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output.
Martin v. Löwis <martin@v.loewis.de>
parents: 26374
diff changeset
847 PyMem_FREE(buf);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
848 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
849 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
850
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
851 str[0] = (Py_UNICODE) code;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
852 return PyUnicode_FromUnicode(str, 1);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
853 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
854
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
855 /* XXX Add doc strings. */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
856
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
857 static PyMethodDef unicodedata_functions[] = {
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
858 {"decimal", unicodedata_decimal, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
859 {"digit", unicodedata_digit, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
860 {"numeric", unicodedata_numeric, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
861 {"category", unicodedata_category, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
862 {"bidirectional", unicodedata_bidirectional, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
863 {"combining", unicodedata_combining, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
864 {"mirrored", unicodedata_mirrored, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
865 {"decomposition",unicodedata_decomposition, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
866 {"name", unicodedata_name, METH_VARARGS},
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
867 {"lookup", unicodedata_lookup, METH_VARARGS},
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
868 {"normalize", unicodedata_normalize, METH_VARARGS},
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
869 {NULL, NULL} /* sentinel */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
870 };
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
871
23832
ed1ba37a46b7 Patch #568124: Add doc string macros.
Martin v. Löwis <martin@v.loewis.de>
parents: 23810
diff changeset
872 PyDoc_STRVAR(unicodedata_docstring, "unicode character database");
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
873
24363
979ff5d0ae07 Replace DL_IMPORT with PyMODINIT_FUNC and remove "/export:init..." link
Mark Hammond <mhammond@skippinet.com.au>
parents: 23832
diff changeset
874 PyMODINIT_FUNC
13651
8a063423e0b7 Bunch of minor ANSIfications: 'void initfunc()' -> 'void initfunc(void)',
Thomas Wouters <thomas@python.org>
parents: 12196
diff changeset
875 initunicodedata(void)
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
876 {
22781
ec67fce3fa3c Remove direct manipulation of the module dict.
Fred Drake <fdrake@acm.org>
parents: 22114
diff changeset
877 PyObject *m, *v;
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
878
16977
0b4c57a25977 Be a bit more strict in setting up the export of the C API for this
Fred Drake <fdrake@acm.org>
parents: 16666
diff changeset
879 m = Py_InitModule3(
0b4c57a25977 Be a bit more strict in setting up the export of the C API for this
Fred Drake <fdrake@acm.org>
parents: 16666
diff changeset
880 "unicodedata", unicodedata_functions, unicodedata_docstring);
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
881 if (!m)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
882 return;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
883
26397
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26380
diff changeset
884 PyModule_AddStringConstant(m, "unidata_version", UNIDATA_VERSION);
b27c1cad77ea Add unidata_version. Bump generator version number.
Martin v. Löwis <martin@v.loewis.de>
parents: 26380
diff changeset
885
16200
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
886 /* Export C API */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent
Fredrik Lundh <fredrik@pythonware.com>
parents: 16091
diff changeset
887 v = PyCObject_FromVoidPtr((void *) &hashAPI, NULL);
22781
ec67fce3fa3c Remove direct manipulation of the module dict.
Fred Drake <fdrake@acm.org>
parents: 22114
diff changeset
888 if (v != NULL)
ec67fce3fa3c Remove direct manipulation of the module dict.
Fred Drake <fdrake@acm.org>
parents: 22114
diff changeset
889 PyModule_AddObject(m, "ucnhash_CAPI", v);
11981
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base.
Guido van Rossum <guido@python.org>
parents:
diff changeset
890 }
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
891
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
892 /*
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
893 Local variables:
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
894 c-basic-offset: 4
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents: 26376
diff changeset
895 indent-tabs-mode: nil
26370
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
896 End:
9e489a0df7b5 Patch #626548: Support Hangul syllable names.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
897 */