cpython-lru_cache-weakref: Modules/unicodedata.c annotate

annotate Modules/unicodedata.c @ 80661:94de77bd0b4b 2.7

Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.

author	Andrew Svetlov <andrew.svetlov@gmail.com>
date	Thu, 29 Nov 2012 19:51:28 +0200
parents	60d80880bf88
children	e9d59cca31fc

rev	line source
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1 /* ------------------------------------------------------------------------
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	2
59926 0f42f34c16e8 Update the version number of the Unicode Database in a few more places. Ezio Melotti <ezio.melotti@gmail.com> parents: 59323 diff changeset	3 unicodedata -- Provides access to the Unicode 5.2 data base.
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	4
59926 0f42f34c16e8 Update the version number of the Unicode Database in a few more places. Ezio Melotti <ezio.melotti@gmail.com> parents: 59323 diff changeset	5 Data was extracted from the Unicode 5.2 UnicodeData.txt file.
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	6
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14818 diff changeset	7 Written by Marc-Andre Lemburg (mal@lemburg.com).
96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14818 diff changeset	8 Modified for Python 2.0 by Fredrik Lundh (fredrik@pythonware.com)
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	9 Modified by Martin v. Löwis (martin@v.loewis.de)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	10
14835 96d31a143857 unicode database compression, step 2: Fredrik Lundh <fredrik@pythonware.com> parents: 14818 diff changeset	11 Copyright (c) Corporation for National Research Initiatives.
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	12
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	13 ------------------------------------------------------------------------ */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	14
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	15 #include "Python.h"
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	16 #include "ucnhash.h"
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	17 #include "structmember.h"
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	18
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	19 /* character properties */
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	20
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	21 typedef struct {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	22 const unsigned char category; /* index into
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	23 _PyUnicode_CategoryNames */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	24 const unsigned char combining; /* combining class value 0 - 255 */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	25 const unsigned char bidirectional; /* index into
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	26 _PyUnicode_BidirectionalNames */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	27 const unsigned char mirrored; /* true if mirrored in bidir mode */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	28 const unsigned char east_asian_width; /* index into
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	29 _PyUnicode_EastAsianWidth */
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	30 const unsigned char normalization_quick_check; /* see is_normalized() */
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	31 } _PyUnicode_DatabaseRecord;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	32
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	33 typedef struct change_record {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	34 /* sequence of fields should be the same as in merge_old_version */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	35 const unsigned char bidir_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	36 const unsigned char category_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	37 const unsigned char decimal_changed;
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47527 diff changeset	38 const unsigned char mirrored_changed;
56406 a9864417286b #1571184: makeunicodedata.py now generates the functions _PyUnicode_ToNumeric, Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53553 diff changeset	39 const double numeric_changed;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	40 } change_record;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	41
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	42 /* data file generated by Tools/unicode/makeunicodedata.py */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	43 #include "unicodedata_db.h"
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	44
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	45 static const _PyUnicode_DatabaseRecord*
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	46 _getrecord_ex(Py_UCS4 code)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	47 {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	48 int index;
28100 54b35da9879e Fix SF bug #694816, remove comparison of unsigned value < 0 Neal Norwitz <nnorwitz@gmail.com> parents: 26571 diff changeset	49 if (code >= 0x110000)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	50 index = 0;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	51 else {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	52 index = index1[(code>>SHIFT)];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	53 index = index2[(index<<SHIFT)+(code&((1<<SHIFT)-1))];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	54 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	55
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	56 return &_PyUnicode_Database_Records[index];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	57 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	58
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	59 /* ------------- Previous-version API ------------------------------------- */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	60 typedef struct previous_version {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	61 PyObject_HEAD
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	62 const char *name;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	63 const change_record* (*getrecord)(Py_UCS4);
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	64 Py_UCS4 (*normalization)(Py_UCS4);
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	65 } PreviousDBVersion;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	66
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	67 #define get_old_record(self, v) ((((PreviousDBVersion*)self)->getrecord)(v))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	68
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	69 static PyMemberDef DB_members[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	70 {"unidata_version", T_STRING, offsetof(PreviousDBVersion, name), READONLY},
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	71 {NULL}
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	72 };
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	73
40319 7059df7c15df Replace C++ comment with C comment (fixes SF bug #1593525). Walter Dörwald <walter@livinglogic.de> parents: 39453 diff changeset	74 /* forward declaration */
36927 3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	75 static PyTypeObject UCD_Type;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	76
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	77 static PyObject*
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	78 new_previous_version(const charname, const change_record (*getrecord)(Py_UCS4),
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	79 Py_UCS4 (*normalization)(Py_UCS4))
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	80 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	81 PreviousDBVersion *self;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	82 self = PyObject_New(PreviousDBVersion, &UCD_Type);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	83 if (self == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	84 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	85 self->name = name;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	86 self->getrecord = getrecord;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	87 self->normalization = normalization;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	88 return (PyObject*)self;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	89 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	90
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	91
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	92 static Py_UCS4 getuchar(PyUnicodeObject *obj)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	93 {
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	94 Py_UNICODE *v = PyUnicode_AS_UNICODE(obj);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	95
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	96 if (PyUnicode_GET_SIZE(obj) == 1)
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	97 return *v;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	98 #ifndef Py_UNICODE_WIDE
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	99 else if ((PyUnicode_GET_SIZE(obj) == 2) &&
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	100 (0xD800 <= v[0] && v[0] <= 0xDBFF) &&
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	101 (0xDC00 <= v[1] && v[1] <= 0xDFFF))
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	102 return (((v[0] & 0x3FF)<<10) \| (v[1] & 0x3FF)) + 0x10000;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	103 #endif
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	104 PyErr_SetString(PyExc_TypeError,
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	105 "need a single Unicode character as parameter");
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	106 return (Py_UCS4)-1;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	107 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	108
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	109 /* --- Module API --------------------------------------------------------- */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	110
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	111 PyDoc_STRVAR(unicodedata_decimal__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	112 "decimal(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	113 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	114 Returns the decimal value assigned to the Unicode character unichr\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	115 as integer. If no such value is defined, default is returned, or, if\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	116 not given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	117
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	118 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	119 unicodedata_decimal(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	120 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	121 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	122 PyObject *defobj = NULL;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	123 int have_old = 0;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	124 long rc;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	125 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	126
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	127 if (!PyArg_ParseTuple(args, "O!\|O:decimal", &PyUnicode_Type, &v, &defobj))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	128 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	129 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	130 if (c == (Py_UCS4)-1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	131 return NULL;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	132
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	133 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	134 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	135 if (old->category_changed == 0) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	136 /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	137 have_old = 1;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	138 rc = -1;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	139 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	140 else if (old->decimal_changed != 0xFF) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	141 have_old = 1;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	142 rc = old->decimal_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	143 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	144 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	145
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	146 if (!have_old)
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	147 rc = Py_UNICODE_TODECIMAL(c);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	148 if (rc < 0) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	149 if (defobj == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	150 PyErr_SetString(PyExc_ValueError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	151 "not a decimal");
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	152 return NULL;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	153 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	154 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	155 Py_INCREF(defobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	156 return defobj;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	157 }
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	158 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	159 return PyInt_FromLong(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	160 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	161
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	162 PyDoc_STRVAR(unicodedata_digit__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	163 "digit(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	164 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	165 Returns the digit value assigned to the Unicode character unichr as\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	166 integer. If no such value is defined, default is returned, or, if\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	167 not given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	168
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	169 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	170 unicodedata_digit(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	171 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	172 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	173 PyObject *defobj = NULL;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	174 long rc;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	175 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	176
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	177 if (!PyArg_ParseTuple(args, "O!\|O:digit", &PyUnicode_Type, &v, &defobj))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	178 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	179 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	180 if (c == (Py_UCS4)-1)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	181 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	182 rc = Py_UNICODE_TODIGIT(c);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	183 if (rc < 0) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	184 if (defobj == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	185 PyErr_SetString(PyExc_ValueError, "not a digit");
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	186 return NULL;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	187 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	188 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	189 Py_INCREF(defobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	190 return defobj;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	191 }
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	192 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	193 return PyInt_FromLong(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	194 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	195
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	196 PyDoc_STRVAR(unicodedata_numeric__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	197 "numeric(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	198 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	199 Returns the numeric value assigned to the Unicode character unichr\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	200 as float. If no such value is defined, default is returned, or, if\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	201 not given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	202
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	203 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	204 unicodedata_numeric(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	205 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	206 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	207 PyObject *defobj = NULL;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	208 int have_old = 0;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	209 double rc;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	210 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	211
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	212 if (!PyArg_ParseTuple(args, "O!\|O:numeric", &PyUnicode_Type, &v, &defobj))
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	213 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	214 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	215 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	216 return NULL;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	217
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	218 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	219 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	220 if (old->category_changed == 0) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	221 /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	222 have_old = 1;
38422 dfbd457ca67a Patch 1494554: Update numeric properties to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 37780 diff changeset	223 rc = -1.0;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	224 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	225 else if (old->decimal_changed != 0xFF) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	226 have_old = 1;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	227 rc = old->decimal_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	228 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	229 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	230
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	231 if (!have_old)
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	232 rc = Py_UNICODE_TONUMERIC(c);
38422 dfbd457ca67a Patch 1494554: Update numeric properties to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 37780 diff changeset	233 if (rc == -1.0) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	234 if (defobj == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	235 PyErr_SetString(PyExc_ValueError, "not a numeric character");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	236 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	237 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	238 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	239 Py_INCREF(defobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	240 return defobj;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	241 }
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	242 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	243 return PyFloat_FromDouble(rc);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	244 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	245
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	246 PyDoc_STRVAR(unicodedata_category__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	247 "category(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	248 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	249 Returns the general category assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	250 unichr as string.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	251
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	252 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	253 unicodedata_category(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	254 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	255 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	256 int index;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	257 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	258
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	259 if (!PyArg_ParseTuple(args, "O!:category",
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	260 &PyUnicode_Type, &v))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	261 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	262 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	263 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	264 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	265 index = (int) _getrecord_ex(c)->category;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	266 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	267 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	268 if (old->category_changed != 0xFF)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	269 index = old->category_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	270 }
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	271 return PyString_FromString(_PyUnicode_CategoryNames[index]);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	272 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	273
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	274 PyDoc_STRVAR(unicodedata_bidirectional__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	275 "bidirectional(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	276 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	277 Returns the bidirectional category assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	278 unichr as string. If no such value is defined, an empty string is\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	279 returned.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	280
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	281 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	282 unicodedata_bidirectional(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	283 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	284 PyUnicodeObject *v;
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	285 int index;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	286 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	287
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	288 if (!PyArg_ParseTuple(args, "O!:bidirectional",
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	289 &PyUnicode_Type, &v))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	290 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	291 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	292 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	293 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	294 index = (int) _getrecord_ex(c)->bidirectional;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	295 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	296 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	297 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	298 index = 0; /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	299 else if (old->bidir_changed != 0xFF)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	300 index = old->bidir_changed;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	301 }
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	302 return PyString_FromString(_PyUnicode_BidirectionalNames[index]);
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	303 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	304
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	305 PyDoc_STRVAR(unicodedata_combining__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	306 "combining(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	307 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	308 Returns the canonical combining class assigned to the Unicode\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	309 character unichr as integer. Returns 0 if no combining class is\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	310 defined.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	311
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	312 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	313 unicodedata_combining(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	314 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	315 PyUnicodeObject *v;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	316 int index;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	317 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	318
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	319 if (!PyArg_ParseTuple(args, "O!:combining",
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	320 &PyUnicode_Type, &v))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	321 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	322 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	323 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	324 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	325 index = (int) _getrecord_ex(c)->combining;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	326 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	327 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	328 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	329 index = 0; /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	330 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	331 return PyInt_FromLong(index);
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	332 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	333
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	334 PyDoc_STRVAR(unicodedata_mirrored__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	335 "mirrored(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	336 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	337 Returns the mirrored property assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	338 unichr as integer. Returns 1 if the character has been identified as\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	339 a \"mirrored\" character in bidirectional text, 0 otherwise.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	340
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	341 static PyObject *
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	342 unicodedata_mirrored(PyObject self, PyObject args)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	343 {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	344 PyUnicodeObject *v;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	345 int index;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	346 Py_UCS4 c;
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	347
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	348 if (!PyArg_ParseTuple(args, "O!:mirrored",
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	349 &PyUnicode_Type, &v))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	350 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	351 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	352 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	353 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	354 index = (int) _getrecord_ex(c)->mirrored;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	355 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	356 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	357 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	358 index = 0; /* unassigned */
49124 c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47527 diff changeset	359 else if (old->mirrored_changed != 0xFF)
c37c7dbd7880 Issue #3811: The Unicode database was updated to 5.1. Martin v. Löwis <martin@v.loewis.de> parents: 47527 diff changeset	360 index = old->mirrored_changed;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	361 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	362 return PyInt_FromLong(index);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	363 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	364
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	365 PyDoc_STRVAR(unicodedata_east_asian_width__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	366 "east_asian_width(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	367 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	368 Returns the east asian width assigned to the Unicode character\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	369 unichr as string.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	370
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	371 static PyObject *
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	372 unicodedata_east_asian_width(PyObject self, PyObject args)
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	373 {
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	374 PyUnicodeObject *v;
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	375 int index;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	376 Py_UCS4 c;
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	377
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	378 if (!PyArg_ParseTuple(args, "O!:east_asian_width",
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	379 &PyUnicode_Type, &v))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	380 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	381 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	382 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	383 return NULL;
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	384 index = (int) _getrecord_ex(c)->east_asian_width;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	385 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	386 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	387 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	388 index = 0; /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	389 }
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	390 return PyString_FromString(_PyUnicode_EastAsianWidthNames[index]);
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	391 }
024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	392
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	393 PyDoc_STRVAR(unicodedata_decomposition__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	394 "decomposition(unichr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	395 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	396 Returns the character decomposition mapping assigned to the Unicode\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	397 character unichr as string. An empty string is returned in case no\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	398 such mapping is defined.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	399
32951 024ff0ab4588 SF #989185: Drop unicode.iswide() and unicode.width() and add Hye-Shik Chang <hyeshik@gmail.com> parents: 32680 diff changeset	400 static PyObject *
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	401 unicodedata_decomposition(PyObject self, PyObject args)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	402 {
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	403 PyUnicodeObject *v;
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	404 char decomp[256];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	405 int code, index, count, i;
39203 0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	406 unsigned int prefix_index;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	407 Py_UCS4 c;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	408
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	409 if (!PyArg_ParseTuple(args, "O!:decomposition",
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	410 &PyUnicode_Type, &v))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	411 return NULL;
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	412 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	413 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	414 return NULL;
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	415
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	416 code = (int)c;
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	417
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	418 if (self) {
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	419 const change_record *old = get_old_record(self, c);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	420 if (old->category_changed == 0)
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	421 return PyString_FromString(""); /* unassigned */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	422 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	423
25852 1f5f6481886d Update to Unicode 3.2 database. Martin v. Löwis <martin@v.loewis.de> parents: 24363 diff changeset	424 if (code < 0 \|\| code >= 0x110000)
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	425 index = 0;
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	426 else {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	427 index = decomp_index1[(code>>DECOMP_SHIFT)];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	428 index = decomp_index2[(index<<DECOMP_SHIFT)+
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	429 (code&((1<<DECOMP_SHIFT)-1))];
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	430 }
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	431
21112 38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf. Tim Peters <tim.peters@gmail.com> parents: 18752 diff changeset	432 /* high byte is number of hex bytes (usually one or two), low byte
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	433 is prefix code (from*/
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	434 count = decomp_data[index] >> 8;
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	435
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	436 /* XXX: could allocate the PyString up front instead
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	437 (strlen(prefix) + 5 * count + 1 bytes) */
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	438
39203 0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	439 /* Based on how index is calculated above and decomp_data is generated
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	440 from Tools/unicode/makeunicodedata.py, it should not be possible
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	441 to overflow decomp_prefix. */
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	442 prefix_index = decomp_data[index] & 255;
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	443 assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix)));
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	444
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	445 /* copy prefix */
39203 0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	446 i = strlen(decomp_prefix[prefix_index]);
0b0cc7a49682 No functional change. Add comment and assert to describe why there cannot be overflow which was reported by Klocwork. Discussed on python-dev Neal Norwitz <nnorwitz@gmail.com> parents: 38422 diff changeset	447 memcpy(decomp, decomp_prefix[prefix_index], i);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	448
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	449 while (count-- > 0) {
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	450 if (i)
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	451 decomp[i++] = ' ';
21112 38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf. Tim Peters <tim.peters@gmail.com> parents: 18752 diff changeset	452 assert((size_t)i < sizeof(decomp));
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf. Tim Peters <tim.peters@gmail.com> parents: 18752 diff changeset	453 PyOS_snprintf(decomp + i, sizeof(decomp) - i, "%04X",
38d0bd0fa7f2 unicodedata_decomposition(): sprintf -> PyOS_snprintf. Tim Peters <tim.peters@gmail.com> parents: 18752 diff changeset	454 decomp_data[++index]);
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	455 i += strlen(decomp + i);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	456 }
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	457
16090 8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	458 decomp[i] = '\0';
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	459
8ac40d48a581 compress unicode decomposition tables (this saves another 55k) Fredrik Lundh <fredrik@pythonware.com> parents: 14835 diff changeset	460 return PyString_FromString(decomp);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	461 }
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	462
37780 b95b5c33cb3a No reason to export get_decomp_record, make static Neal Norwitz <nnorwitz@gmail.com> parents: 37664 diff changeset	463 static void
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	464 get_decomp_record(PyObject self, Py_UCS4 code, int index, int prefix, int count)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	465 {
28100 54b35da9879e Fix SF bug #694816, remove comparison of unsigned value < 0 Neal Norwitz <nnorwitz@gmail.com> parents: 26571 diff changeset	466 if (code >= 0x110000) {
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	467 *index = 0;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	468 } else if (self && get_old_record(self, code)->category_changed==0) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	469 /* unassigned in old version */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	470 *index = 0;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	471 }
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	472 else {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	473 *index = decomp_index1[(code>>DECOMP_SHIFT)];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	474 index = decomp_index2[(index<<DECOMP_SHIFT)+
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	475 (code&((1<<DECOMP_SHIFT)-1))];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	476 }
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	477
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	478 /* high byte is number of hex bytes (usually one or two), low byte
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	479 is prefix code (from*/
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	480 count = decomp_data[index] >> 8;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	481 prefix = decomp_data[index] & 255;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	482
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	483 (*index)++;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	484 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	485
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	486 #define SBase 0xAC00
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	487 #define LBase 0x1100
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	488 #define VBase 0x1161
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	489 #define TBase 0x11A7
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	490 #define LCount 19
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	491 #define VCount 21
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	492 #define TCount 28
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	493 #define NCount (VCount*TCount)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	494 #define SCount (LCount*NCount)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	495
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	496 static PyObject*
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	497 nfd_nfkd(PyObject self, PyObject input, int k)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	498 {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	499 PyObject *result;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	500 Py_UNICODE i, end, *o;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	501 /* Longest decomposition in Unicode 3.2: U+FDFA */
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	502 Py_UNICODE stack[20];
37664 d13ded9165ec Support NFD of very long strings. Martin v. Löwis <martin@v.loewis.de> parents: 37556 diff changeset	503 Py_ssize_t space, isize;
d13ded9165ec Support NFD of very long strings. Martin v. Löwis <martin@v.loewis.de> parents: 37556 diff changeset	504 int index, prefix, count, stackptr;
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	505 unsigned char prev, cur;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	506
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	507 stackptr = 0;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	508 isize = PyUnicode_GET_SIZE(input);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	509 /* Overallocate atmost 10 characters. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	510 space = (isize > 10 ? 10 : isize) + isize;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	511 result = PyUnicode_FromUnicode(NULL, space);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	512 if (!result)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	513 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	514 i = PyUnicode_AS_UNICODE(input);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	515 end = i + isize;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	516 o = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	517
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	518 while (i < end) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	519 stack[stackptr++] = *i++;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	520 while(stackptr) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	521 Py_UNICODE code = stack[--stackptr];
30867 04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676. Martin v. Löwis <martin@v.loewis.de> parents: 28100 diff changeset	522 /* Hangul Decomposition adds three characters in
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676. Martin v. Löwis <martin@v.loewis.de> parents: 28100 diff changeset	523 a single step, so we need atleast that much room. */
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676. Martin v. Löwis <martin@v.loewis.de> parents: 28100 diff changeset	524 if (space < 3) {
37664 d13ded9165ec Support NFD of very long strings. Martin v. Löwis <martin@v.loewis.de> parents: 37556 diff changeset	525 Py_ssize_t newsize = PyString_GET_SIZE(result) + 10;
30867 04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676. Martin v. Löwis <martin@v.loewis.de> parents: 28100 diff changeset	526 space += 10;
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676. Martin v. Löwis <martin@v.loewis.de> parents: 28100 diff changeset	527 if (PyUnicode_Resize(&result, newsize) == -1)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	528 return NULL;
30867 04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676. Martin v. Löwis <martin@v.loewis.de> parents: 28100 diff changeset	529 o = PyUnicode_AS_UNICODE(result) + newsize - space;
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	530 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	531 /* Hangul Decomposition. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	532 if (SBase <= code && code < (SBase+SCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	533 int SIndex = code - SBase;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	534 int L = LBase + SIndex / NCount;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	535 int V = VBase + (SIndex % NCount) / TCount;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	536 int T = TBase + SIndex % TCount;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	537 *o++ = L;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	538 *o++ = V;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	539 space -= 2;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	540 if (T != TBase) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	541 *o++ = T;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	542 space --;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	543 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	544 continue;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	545 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	546 /* normalization changes */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	547 if (self) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	548 Py_UCS4 value = ((PreviousDBVersion*)self)->normalization(code);
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	549 if (value != 0) {
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	550 stack[stackptr++] = value;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	551 continue;
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	552 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	553 }
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	554
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	555 /* Other decompositions. */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	556 get_decomp_record(self, code, &index, &prefix, &count);
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	557
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	558 /* Copy character if it is not decomposable, or has a
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	559 compatibility decomposition, but we do NFD. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	560 if (!count \|\| (prefix && !k)) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	561 *o++ = code;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	562 space--;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	563 continue;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	564 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	565 /* Copy decomposition onto the stack, in reverse
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	566 order. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	567 while(count) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	568 code = decomp_data[index + (--count)];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	569 stack[stackptr++] = code;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	570 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	571 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	572 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	573
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	574 /* Drop overallocation. Cannot fail. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	575 PyUnicode_Resize(&result, PyUnicode_GET_SIZE(result) - space);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	576
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	577 /* Sort canonically. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	578 i = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	579 prev = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	580 end = i + PyUnicode_GET_SIZE(result);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	581 for (i++; i < end; i++) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	582 cur = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	583 if (prev == 0 \|\| cur == 0 \|\| prev <= cur) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	584 prev = cur;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	585 continue;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	586 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	587 /* Non-canonical order. Need to switch i with previous. /
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	588 o = i - 1;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	589 while (1) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	590 Py_UNICODE tmp = o[1];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	591 o[1] = o[0];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	592 o[0] = tmp;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	593 o--;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	594 if (o < PyUnicode_AS_UNICODE(result))
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	595 break;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	596 prev = _getrecord_ex(*o)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	597 if (prev == 0 \|\| prev <= cur)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	598 break;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	599 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	600 prev = _getrecord_ex(*i)->combining;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	601 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	602 return result;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	603 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	604
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	605 static int
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	606 find_nfc_index(PyObject self, struct reindex nfc, Py_UNICODE code)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	607 {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	608 int index;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	609 for (index = 0; nfc[index].start; index++) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	610 int start = nfc[index].start;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	611 if (code < start)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	612 return -1;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	613 if (code <= start + nfc[index].count) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	614 int delta = code - start;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	615 return nfc[index].index + delta;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	616 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	617 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	618 return -1;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	619 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	620
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	621 static PyObject*
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	622 nfc_nfkc(PyObject self, PyObject input, int k)
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	623 {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	624 PyObject *result;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	625 Py_UNICODE i, i1, o, end;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	626 int f,l,index,index1,comb;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	627 Py_UNICODE code;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	628 Py_UNICODE *skipped[20];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	629 int cskipped = 0;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	630
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	631 result = nfd_nfkd(self, input, k);
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	632 if (!result)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	633 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	634
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	635 /* We are going to modify result in-place.
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	636 If nfd_nfkd is changed to sometimes return the input,
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	637 this code needs to be reviewed. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	638 assert(result != input);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	639
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	640 i = PyUnicode_AS_UNICODE(result);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	641 end = i + PyUnicode_GET_SIZE(result);
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	642 o = PyUnicode_AS_UNICODE(result);
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	643
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	644 again:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	645 while (i < end) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	646 for (index = 0; index < cskipped; index++) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	647 if (skipped[index] == i) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	648 /* *i character is skipped.
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	649 Remove from list. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	650 skipped[index] = skipped[cskipped-1];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	651 cskipped--;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	652 i++;
26571 cb8cfadf75a2 Remove C++ comment. Martin v. Löwis <martin@v.loewis.de> parents: 26397 diff changeset	653 goto again; /* continue while */
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	654 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	655 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	656 /* Hangul Composition. We don't need to check for <LV,T>
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	657 pairs, since we always have decomposed data. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	658 if (LBase <= i && i < (LBase+LCount) &&
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	659 i + 1 < end &&
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	660 VBase <= i[1] && i[1] <= (VBase+VCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	661 int LIndex, VIndex;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	662 LIndex = i[0] - LBase;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	663 VIndex = i[1] - VBase;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	664 code = SBase + (LIndexVCount+VIndex)TCount;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	665 i+=2;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	666 if (i < end &&
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	667 TBase <= i && i <= (TBase+TCount)) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	668 code += *i-TBase;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	669 i++;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	670 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	671 *o++ = code;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	672 continue;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	673 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	674
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	675 f = find_nfc_index(self, nfc_first, *i);
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	676 if (f == -1) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	677 o++ = i++;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	678 continue;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	679 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	680 /* Find next unblocked character. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	681 i1 = i+1;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	682 comb = 0;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	683 while (i1 < end) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	684 int comb1 = _getrecord_ex(*i1)->combining;
67190 52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	685 if (comb) {
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	686 if (comb1 == 0)
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	687 break;
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	688 if (comb >= comb1) {
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	689 /* Character is blocked. */
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	690 i1++;
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	691 continue;
52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	692 }
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	693 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	694 l = find_nfc_index(self, nfc_last, *i1);
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	695 /* i1 cannot be combined with i. If *i1
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	696 is a starter, we don't need to look further.
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	697 Otherwise, record the combining class. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	698 if (l == -1) {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	699 not_combinable:
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	700 if (comb1 == 0)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	701 break;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	702 comb = comb1;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	703 i1++;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	704 continue;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	705 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	706 index = f*TOTAL_LAST + l;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	707 index1 = comp_index[index >> COMP_SHIFT];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	708 code = comp_data[(index1<<COMP_SHIFT)+
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	709 (index&((1<<COMP_SHIFT)-1))];
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	710 if (code == 0)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	711 goto not_combinable;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	712
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	713 /* Replace the original character. */
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	714 *i = code;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	715 /* Mark the second character unused. */
67190 52a861746129 Merged revisions 87541,87543 via svnmerge from Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 61465 diff changeset	716 assert(cskipped < 20);
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	717 skipped[cskipped++] = i1;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	718 i1++;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	719 f = find_nfc_index(self, nfc_first, *i);
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	720 if (f == -1)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	721 break;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	722 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	723 o++ = i++;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	724 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	725 if (o != end)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	726 PyUnicode_Resize(&result, o - PyUnicode_AS_UNICODE(result));
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	727 return result;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	728 }
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	729
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	730 /* Return 1 if the input is certainly normalized, 0 if it might not be. */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	731 static int
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	732 is_normalized(PyObject self, PyObject input, int nfc, int k)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	733 {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	734 Py_UNICODE i, end;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	735 unsigned char prev_combining = 0, quickcheck_mask;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	736
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	737 /* An older version of the database is requested, quickchecks must be
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	738 disabled. */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	739 if (self != NULL)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	740 return 0;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	741
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	742 /* The two quickcheck bits at this shift mean 0=Yes, 1=Maybe, 2=No,
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	743 as described in http://unicode.org/reports/tr15/#Annex8. */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	744 quickcheck_mask = 3 << ((nfc ? 4 : 0) + (k ? 2 : 0));
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	745
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	746 i = PyUnicode_AS_UNICODE(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	747 end = i + PyUnicode_GET_SIZE(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	748 while (i < end) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	749 const _PyUnicode_DatabaseRecord record = _getrecord_ex(i++);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	750 unsigned char combining = record->combining;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	751 unsigned char quickcheck = record->normalization_quick_check;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	752
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	753 if (quickcheck & quickcheck_mask)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	754 return 0; /* this string might need normalization */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	755 if (combining && prev_combining > combining)
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	756 return 0; /* non-canonical sort order, not normalized */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	757 prev_combining = combining;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	758 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	759 return 1; /* certainly normalized */
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	760 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	761
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	762 PyDoc_STRVAR(unicodedata_normalize__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	763 "normalize(form, unistr)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	764 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	765 Return the normal form 'form' for the Unicode string unistr. Valid\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	766 values for form are 'NFC', 'NFKC', 'NFD', and 'NFKD'.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	767
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	768 static PyObject*
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	769 unicodedata_normalize(PyObject self, PyObject args)
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	770 {
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	771 char *form;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	772 PyObject *input;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	773
32680 d3d426483d71 Fix typo. Hye-Shik Chang <hyeshik@gmail.com> parents: 31930 diff changeset	774 if(!PyArg_ParseTuple(args, "sO!:normalize",
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	775 &form, &PyUnicode_Type, &input))
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	776 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	777
31930 f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	778 if (PyUnicode_GetSize(input) == 0) {
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	779 /* Special case empty input strings, since resizing
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	780 them later would cause internal errors. */
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	781 Py_INCREF(input);
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	782 return input;
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	783 }
f8c0ac5bef02 Special case normalization of empty strings. Fixes #924361. Martin v. Löwis <martin@v.loewis.de> parents: 30867 diff changeset	784
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	785 if (strcmp(form, "NFC") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	786 if (is_normalized(self, input, 1, 0)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	787 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	788 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	789 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	790 return nfc_nfkc(self, input, 0);
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	791 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	792 if (strcmp(form, "NFKC") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	793 if (is_normalized(self, input, 1, 1)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	794 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	795 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	796 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	797 return nfc_nfkc(self, input, 1);
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	798 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	799 if (strcmp(form, "NFD") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	800 if (is_normalized(self, input, 0, 0)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	801 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	802 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	803 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	804 return nfd_nfkd(self, input, 0);
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	805 }
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	806 if (strcmp(form, "NFKD") == 0) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	807 if (is_normalized(self, input, 0, 1)) {
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	808 Py_INCREF(input);
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	809 return input;
019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	810 }
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	811 return nfd_nfkd(self, input, 1);
53553 019244f4b0d0 Issue #1734234: Massively speedup `unicodedata.normalize()` when the Antoine Pitrou <solipsis@pitrou.net> parents: 49124 diff changeset	812 }
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	813 PyErr_SetString(PyExc_ValueError, "invalid normalization form");
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	814 return NULL;
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	815 }
1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	816
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	817 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	818 /* unicode character name tables */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	819
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	820 /* data file generated by Tools/unicode/makeunicodedata.py */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	821 #include "unicodename_db.h"
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	822
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	823 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	824 /* database code (cut and pasted from the unidb package) */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	825
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	826 static unsigned long
16666 e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS Fredrik Lundh <fredrik@pythonware.com> parents: 16662 diff changeset	827 _gethash(const char *s, int len, int scale)
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	828 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	829 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	830 unsigned long h = 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	831 unsigned long ix;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	832 for (i = 0; i < len; i++) {
72657 60d80880bf88 Remove all other uses of the C tolower()/toupper() which could break with a Turkish locale. Antoine Pitrou <solipsis@pitrou.net> parents: 68256 diff changeset	833 h = (h * scale) + (unsigned char) Py_TOUPPER(Py_CHARMASK(s[i]));
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	834 ix = h & 0xff000000;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	835 if (ix)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	836 h = (h ^ ((ix>>24) & 0xff)) & 0x00ffffff;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	837 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	838 return h;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	839 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	840
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	841 static char *hangul_syllables[][3] = {
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	842 { "G", "A", "" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	843 { "GG", "AE", "G" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	844 { "N", "YA", "GG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	845 { "D", "YAE", "GS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	846 { "DD", "EO", "N", },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	847 { "R", "E", "NJ" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	848 { "M", "YEO", "NH" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	849 { "B", "YE", "D" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	850 { "BB", "O", "L" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	851 { "S", "WA", "LG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	852 { "SS", "WAE", "LM" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	853 { "", "OE", "LB" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	854 { "J", "YO", "LS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	855 { "JJ", "U", "LT" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	856 { "C", "WEO", "LP" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	857 { "K", "WE", "LH" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	858 { "T", "WI", "M" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	859 { "P", "YU", "B" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	860 { "H", "EU", "BS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	861 { 0, "YI", "S" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	862 { 0, "I", "SS" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	863 { 0, 0, "NG" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	864 { 0, 0, "J" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	865 { 0, 0, "C" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	866 { 0, 0, "K" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	867 { 0, 0, "T" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	868 { 0, 0, "P" },
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	869 { 0, 0, "H" }
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	870 };
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	871
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	872 static int
26380 f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	873 is_unified_ideograph(Py_UCS4 code)
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	874 {
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	875 return (
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	876 (0x3400 <= code && code <= 0x4DB5) \|\| /* CJK Ideograph Extension A */
66397 e0eccfa9ad6b Issue #10459: Update CJK character names to Unicode 5.2. Martin v. Löwis <martin@v.loewis.de> parents: 61463 diff changeset	877 (0x4E00 <= code && code <= 0x9FCB) \|\| /* CJK Ideograph, Unicode 5.2 */
e0eccfa9ad6b Issue #10459: Update CJK character names to Unicode 5.2. Martin v. Löwis <martin@v.loewis.de> parents: 61463 diff changeset	878 (0x20000 <= code && code <= 0x2A6D6) \|\| /* CJK Ideograph Extension B */
e0eccfa9ad6b Issue #10459: Update CJK character names to Unicode 5.2. Martin v. Löwis <martin@v.loewis.de> parents: 61463 diff changeset	879 (0x2A700 <= code && code <= 0x2B734)); /* CJK Ideograph Extension C */
26380 f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	880 }
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	881
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	882 static int
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	883 _getucname(PyObject self, Py_UCS4 code, char buffer, int buflen)
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	884 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	885 int offset;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	886 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	887 int word;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	888 unsigned char* w;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	889
36939 03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	890 if (code >= 0x110000)
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	891 return 0;
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	892
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	893 if (self) {
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	894 const change_record *old = get_old_record(self, code);
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	895 if (old->category_changed == 0) {
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	896 /* unassigned */
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	897 return 0;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	898 }
36939 03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	899 }
03ef987043ce Adjust CJK Ideograph range to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36928 diff changeset	900
26374 135bb11215a0 Fix off-by-one error. Martin v. Löwis <martin@v.loewis.de> parents: 26370 diff changeset	901 if (SBase <= code && code < SBase+SCount) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	902 /* Hangul syllable. */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	903 int SIndex = code - SBase;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	904 int L = SIndex / NCount;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	905 int V = (SIndex % NCount) / TCount;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	906 int T = SIndex % TCount;
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	907
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	908 if (buflen < 27)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	909 /* Worst case: HANGUL SYLLABLE <10chars>. */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	910 return 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	911 strcpy(buffer, "HANGUL SYLLABLE ");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	912 buffer += 16;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	913 strcpy(buffer, hangul_syllables[L][0]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	914 buffer += strlen(hangul_syllables[L][0]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	915 strcpy(buffer, hangul_syllables[V][1]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	916 buffer += strlen(hangul_syllables[V][1]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	917 strcpy(buffer, hangul_syllables[T][2]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	918 buffer += strlen(hangul_syllables[T][2]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	919 *buffer = '\0';
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	920 return 1;
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	921 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	922
26380 f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	923 if (is_unified_ideograph(code)) {
26376 0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	924 if (buflen < 28)
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	925 /* Worst case: CJK UNIFIED IDEOGRAPH-20000 */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	926 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	927 sprintf(buffer, "CJK UNIFIED IDEOGRAPH-%X", code);
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	928 return 1;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	929 }
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	930
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	931 /* get offset into phrasebook */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	932 offset = phrasebook_offset1[(code>>phrasebook_shift)];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	933 offset = phrasebook_offset2[(offset<<phrasebook_shift) +
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	934 (code&((1<<phrasebook_shift)-1))];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	935 if (!offset)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	936 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	937
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	938 i = 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	939
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	940 for (;;) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	941 /* get word index */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	942 word = phrasebook[offset] - phrasebook_short;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	943 if (word >= 0) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	944 word = (word << 8) + phrasebook[offset+1];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	945 offset += 2;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	946 } else
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	947 word = phrasebook[offset++];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	948 if (i) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	949 if (i > buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	950 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	951 buffer[i++] = ' ';
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	952 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	953 /* copy word string from lexicon. the last character in the
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	954 word has bit 7 set. the last word in a string ends with
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	955 0x80 */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	956 w = lexicon + lexicon_offset[word];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	957 while (*w < 128) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	958 if (i >= buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	959 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	960 buffer[i++] = *w++;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	961 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	962 if (i >= buflen)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	963 return 0; /* buffer overflow */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	964 buffer[i++] = *w & 127;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	965 if (*w == 128)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	966 break; /* end of word */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	967 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	968
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	969 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	970 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	971
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	972 static int
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	973 _cmpname(PyObject self, int code, const char name, int namelen)
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	974 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	975 /* check if code corresponds to the given name */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	976 int i;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	977 char buffer[NAME_MAXLEN];
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	978 if (!_getucname(self, code, buffer, sizeof(buffer)))
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	979 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	980 for (i = 0; i < namelen; i++) {
72657 60d80880bf88 Remove all other uses of the C tolower()/toupper() which could break with a Turkish locale. Antoine Pitrou <solipsis@pitrou.net> parents: 68256 diff changeset	981 if (Py_TOUPPER(Py_CHARMASK(name[i])) != buffer[i])
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	982 return 0;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	983 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	984 return buffer[namelen] == '\0';
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	985 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	986
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	987 static void
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	988 find_syllable(const char str, int len, int *pos, int count, int column)
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	989 {
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	990 int i, len1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	991 *len = -1;
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	992 for (i = 0; i < count; i++) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	993 char *s = hangul_syllables[i][column];
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	994 len1 = strlen(s);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	995 if (len1 <= *len)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	996 continue;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	997 if (strncmp(str, s, len1) == 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	998 *len = len1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	999 *pos = i;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1000 }
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1001 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1002 if (*len == -1) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1003 *len = 0;
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1004 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1005 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1006
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1007 static int
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1008 _getcode(PyObject* self, const char* name, int namelen, Py_UCS4* code)
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1009 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1010 unsigned int h, v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1011 unsigned int mask = code_size-1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1012 unsigned int i, incr;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1013
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1014 /* Check for hangul syllables. */
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1015 if (strncmp(name, "HANGUL SYLLABLE ", 16) == 0) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1016 int len, L = -1, V = -1, T = -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1017 const char *pos = name + 16;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1018 find_syllable(pos, &len, &L, LCount, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1019 pos += len;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1020 find_syllable(pos, &len, &V, VCount, 1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1021 pos += len;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1022 find_syllable(pos, &len, &T, TCount, 2);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1023 pos += len;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1024 if (L != -1 && V != -1 && T != -1 && pos-name == namelen) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1025 code = SBase + (LVCount+V)*TCount + T;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1026 return 1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1027 }
26376 0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1028 /* Otherwise, it's an illegal syllable name. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1029 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1030 }
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1031
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1032 /* Check for unified ideographs. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1033 if (strncmp(name, "CJK UNIFIED IDEOGRAPH-", 22) == 0) {
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1034 /* Four or five hexdigits must follow. */
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1035 v = 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1036 name += 22;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1037 namelen -= 22;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1038 if (namelen != 4 && namelen != 5)
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1039 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1040 while (namelen--) {
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1041 v *= 16;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1042 if (name >= '0' && name <= '9')
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1043 v += *name - '0';
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1044 else if (name >= 'A' && name <= 'F')
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1045 v += *name - 'A' + 10;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1046 else
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1047 return 0;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1048 name++;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1049 }
26380 f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	1050 if (!is_unified_ideograph(v))
f8a1ba446158 Verify that the code in CJK UNIFIED IDEOGRAPH- actually denotes an ideograph. Martin v. Löwis <martin@v.loewis.de> parents: 26379 diff changeset	1051 return 0;
26376 0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1052 *code = v;
0f2766302ba0 Implement names for CJK unified ideographs. Add name to KeyError output. Martin v. Löwis <martin@v.loewis.de> parents: 26374 diff changeset	1053 return 1;
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1054 }
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1055
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1056 /* the following is the same as python's dictionary lookup, with
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1057 only minor changes. see the makeunicodedata script for more
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1058 details */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1059
16666 e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS Fredrik Lundh <fredrik@pythonware.com> parents: 16662 diff changeset	1060 h = (unsigned int) _gethash(name, namelen, code_magic);
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1061 i = (~h) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1062 v = code_hash[i];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1063 if (!v)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1064 return 0;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1065 if (_cmpname(self, v, name, namelen)) {
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1066 *code = v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1067 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1068 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1069 incr = (h ^ (h >> 3)) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1070 if (!incr)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1071 incr = mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1072 for (;;) {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1073 i = (i + incr) & mask;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1074 v = code_hash[i];
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1075 if (!v)
16662 89c1ccef5ce8 stupid typo (for some reason, this only caused problems on OpenVMS). Fredrik Lundh <fredrik@pythonware.com> parents: 16200 diff changeset	1076 return 0;
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1077 if (_cmpname(self, v, name, namelen)) {
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1078 *code = v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1079 return 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1080 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1081 incr = incr << 1;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1082 if (incr > mask)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1083 incr = incr ^ code_poly;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1084 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1085 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1086
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1087 static const _PyUnicode_Name_CAPI hashAPI =
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1088 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1089 sizeof(_PyUnicode_Name_CAPI),
23810 01e4b698ff7e _Py prefix is verboten for static entry points Andrew MacIntyre <andymac@bullseye.apana.org.au> parents: 22781 diff changeset	1090 _getucname,
16666 e9fcb46c1f2c renamed internal functions to avoid name clashes under OpenVMS Fredrik Lundh <fredrik@pythonware.com> parents: 16662 diff changeset	1091 _getcode
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1092 };
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1093
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1094 /* -------------------------------------------------------------------- */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1095 /* Python bindings */
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1096
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1097 PyDoc_STRVAR(unicodedata_name__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1098 "name(unichr[, default])\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1099 Returns the name assigned to the Unicode character unichr as a\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1100 string. If no name is defined, default is returned, or, if not\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1101 given, ValueError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1102
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1103 static PyObject *
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1104 unicodedata_name(PyObject* self, PyObject* args)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1105 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1106 char name[NAME_MAXLEN];
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	1107 Py_UCS4 c;
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1108
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1109 PyUnicodeObject* v;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1110 PyObject* defobj = NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1111 if (!PyArg_ParseTuple(args, "O!\|O:name", &PyUnicode_Type, &v, &defobj))
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1112 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1113
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	1114 c = getuchar(v);
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	1115 if (c == (Py_UCS4)-1)
ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	1116 return NULL;
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1117
47451 ed471a8009a6 Change all functions that expect one unicode character to accept a pair of Walter Dörwald <walter@livinglogic.de> parents: 47294 diff changeset	1118 if (!_getucname(self, c, name, sizeof(name))) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1119 if (defobj == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1120 PyErr_SetString(PyExc_ValueError, "no such name");
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1121 return NULL;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1122 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1123 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1124 Py_INCREF(defobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1125 return defobj;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1126 }
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1127 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1128
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1129 return Py_BuildValue("s", name);
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1130 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1131
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1132 PyDoc_STRVAR(unicodedata_lookup__doc__,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1133 "lookup(name)\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1134 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1135 Look up character by name. If a character with the\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1136 given name is found, return the corresponding Unicode\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1137 character. If not found, KeyError is raised.");
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1138
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1139 static PyObject *
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1140 unicodedata_lookup(PyObject* self, PyObject* args)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1141 {
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1142 Py_UCS4 code;
42387 85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 42333 diff changeset	1143 Py_UNICODE str[2];
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1144
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1145 char* name;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1146 int namelen;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1147 if (!PyArg_ParseTuple(args, "s#:lookup", &name, &namelen))
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1148 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1149
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1150 if (!_getcode(self, name, namelen, &code)) {
42387 85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 42333 diff changeset	1151 PyErr_Format(PyExc_KeyError, "undefined character name '%s'",
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 42333 diff changeset	1152 name);
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1153 return NULL;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1154 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1155
42386 7e7dce3089e3 Bug #1704793: Raise KeyError if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 40319 diff changeset	1156 #ifndef Py_UNICODE_WIDE
7e7dce3089e3 Bug #1704793: Raise KeyError if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 40319 diff changeset	1157 if (code >= 0x10000) {
42387 85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 42333 diff changeset	1158 str[0] = 0xd800 + ((code - 0x10000) >> 10);
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 42333 diff changeset	1159 str[1] = 0xdc00 + ((code - 0x10000) & 0x3ff);
85b7f40e1c1c Bug #1704793: Return UTF-16 pair if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 42333 diff changeset	1160 return PyUnicode_FromUnicode(str, 2);
42386 7e7dce3089e3 Bug #1704793: Raise KeyError if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 40319 diff changeset	1161 }
7e7dce3089e3 Bug #1704793: Raise KeyError if unicodedata.lookup cannot Martin v. Löwis <martin@v.loewis.de> parents: 40319 diff changeset	1162 #endif
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1163 str[0] = (Py_UNICODE) code;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1164 return PyUnicode_FromUnicode(str, 1);
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1165 }
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1166
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1167 /* XXX Add doc strings. */
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1168
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1169 static PyMethodDef unicodedata_functions[] = {
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1170 {"decimal", unicodedata_decimal, METH_VARARGS, unicodedata_decimal__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1171 {"digit", unicodedata_digit, METH_VARARGS, unicodedata_digit__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1172 {"numeric", unicodedata_numeric, METH_VARARGS, unicodedata_numeric__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1173 {"category", unicodedata_category, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1174 unicodedata_category__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1175 {"bidirectional", unicodedata_bidirectional, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1176 unicodedata_bidirectional__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1177 {"combining", unicodedata_combining, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1178 unicodedata_combining__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1179 {"mirrored", unicodedata_mirrored, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1180 unicodedata_mirrored__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1181 {"east_asian_width", unicodedata_east_asian_width, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1182 unicodedata_east_asian_width__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1183 {"decomposition", unicodedata_decomposition, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1184 unicodedata_decomposition__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1185 {"name", unicodedata_name, METH_VARARGS, unicodedata_name__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1186 {"lookup", unicodedata_lookup, METH_VARARGS, unicodedata_lookup__doc__},
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1187 {"normalize", unicodedata_normalize, METH_VARARGS,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1188 unicodedata_normalize__doc__},
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1189 {NULL, NULL} /* sentinel */
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1190 };
d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1191
36927 3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1192 static PyTypeObject UCD_Type = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1193 /* The ob_type field must be initialized in the module init function
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1194 * to be portable to Windows without using C++. */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1195 PyVarObject_HEAD_INIT(NULL, 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1196 "unicodedata.UCD", /tp_name/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1197 sizeof(PreviousDBVersion), /tp_basicsize/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1198 0, /tp_itemsize/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1199 /* methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1200 (destructor)PyObject_Del, /tp_dealloc/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1201 0, /tp_print/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1202 0, /tp_getattr/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1203 0, /tp_setattr/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1204 0, /tp_compare/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1205 0, /tp_repr/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1206 0, /tp_as_number/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1207 0, /tp_as_sequence/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1208 0, /tp_as_mapping/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1209 0, /tp_hash/
36927 3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1210 0, /tp_call/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1211 0, /tp_str/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1212 PyObject_GenericGetAttr,/tp_getattro/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1213 0, /tp_setattro/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1214 0, /tp_as_buffer/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1215 Py_TPFLAGS_DEFAULT, /tp_flags/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1216 0, /tp_doc/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1217 0, /tp_traverse/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1218 0, /tp_clear/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1219 0, /tp_richcompare/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1220 0, /tp_weaklistoffset/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1221 0, /tp_iter/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1222 0, /tp_iternext/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1223 unicodedata_functions, /tp_methods/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1224 DB_members, /tp_members/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1225 0, /tp_getset/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1226 0, /tp_base/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1227 0, /tp_dict/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1228 0, /tp_descr_get/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1229 0, /tp_descr_set/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1230 0, /tp_dictoffset/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1231 0, /tp_init/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1232 0, /tp_alloc/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1233 0, /tp_new/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1234 0, /tp_free/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1235 0, /tp_is_gc/
3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1236 };
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1237
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1238 PyDoc_STRVAR(unicodedata_docstring,
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1239 "This module provides access to the Unicode Character Database which\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1240 defines character properties for all Unicode characters. The data in\n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1241 this database is based on the UnicodeData.txt file version\n\
59926 0f42f34c16e8 Update the version number of the Unicode Database in a few more places. Ezio Melotti <ezio.melotti@gmail.com> parents: 59323 diff changeset	1242 5.2.0 which is publically available from ftp://ftp.unicode.org/.\n\
34864 34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1243 \n\
34cdfc5dd7ea Fill docstrings for module and functions, extracted from the tex Hye-Shik Chang <hyeshik@gmail.com> parents: 32951 diff changeset	1244 The module uses the same names and symbols as defined by the\n\
59932 684cd7381535 Link specifically to the UCD version 5.2.0. Ezio Melotti <ezio.melotti@gmail.com> parents: 59926 diff changeset	1245 UnicodeData File Format 5.2.0 (see\n\
684cd7381535 Link specifically to the UCD version 5.2.0. Ezio Melotti <ezio.melotti@gmail.com> parents: 59926 diff changeset	1246 http://www.unicode.org/reports/tr44/tr44-4.html).");
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1247
24363 979ff5d0ae07 Replace DL_IMPORT with PyMODINIT_FUNC and remove "/export:init..." link Mark Hammond <mhammond@skippinet.com.au> parents: 23832 diff changeset	1248 PyMODINIT_FUNC
13651 8a063423e0b7 Bunch of minor ANSIfications: 'void initfunc()' -> 'void initfunc(void)', Thomas Wouters <thomas@python.org> parents: 12196 diff changeset	1249 initunicodedata(void)
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1250 {
22781 ec67fce3fa3c Remove direct manipulation of the module dict. Fred Drake <fdrake@acm.org> parents: 22114 diff changeset	1251 PyObject m, v;
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1252
44350 26384b8180c7 #1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available. Christian Heimes <christian@cheimes.de> parents: 42387 diff changeset	1253 Py_TYPE(&UCD_Type) = &PyType_Type;
36927 3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1254
16977 0b4c57a25977 Be a bit more strict in setting up the export of the C API for this Fred Drake <fdrake@acm.org> parents: 16666 diff changeset	1255 m = Py_InitModule3(
0b4c57a25977 Be a bit more strict in setting up the export of the C API for this Fred Drake <fdrake@acm.org> parents: 16666 diff changeset	1256 "unicodedata", unicodedata_functions, unicodedata_docstring);
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1257 if (!m)
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1258 return;
3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1259
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26380 diff changeset	1260 PyModule_AddStringConstant(m, "unidata_version", UNIDATA_VERSION);
36928 11f8b5b8ce20 Fix refcounting bug. Martin v. Löwis <martin@v.loewis.de> parents: 36927 diff changeset	1261 Py_INCREF(&UCD_Type);
36927 3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1262 PyModule_AddObject(m, "UCD", (PyObject*)&UCD_Type);
26397 b27c1cad77ea Add unidata_version. Bump generator version number. Martin v. Löwis <martin@v.loewis.de> parents: 26380 diff changeset	1263
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1264 /* Previous versions */
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1265 v = new_previous_version("3.2.0", get_change_3_2_0, normalization_3_2_0);
48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1266 if (v != NULL)
36927 3266a020a0da Avoid forward-declaring the methods array. Martin v. Löwis <martin@v.loewis.de> parents: 36924 diff changeset	1267 PyModule_AddObject(m, "ucd_3_2_0", v);
36924 48e0a5e39403 Update Unicode database to Unicode 4.1. Martin v. Löwis <martin@v.loewis.de> parents: 36778 diff changeset	1268
16200 3484670c37f2 Move uchhash functionality into unicodedata (after the recent Fredrik Lundh <fredrik@pythonware.com> parents: 16091 diff changeset	1269 /* Export C API */
59998 d11fbc796499 Backported PyCapsule from 3.1, and converted most uses of Larry Hastings <larry@hastings.org> parents: 59932 diff changeset	1270 v = PyCapsule_New((void *)&hashAPI, PyUnicodeData_CAPSULE_NAME, NULL);
22781 ec67fce3fa3c Remove direct manipulation of the module dict. Fred Drake <fdrake@acm.org> parents: 22114 diff changeset	1271 if (v != NULL)
ec67fce3fa3c Remove direct manipulation of the module dict. Fred Drake <fdrake@acm.org> parents: 22114 diff changeset	1272 PyModule_AddObject(m, "ucnhash_CAPI", v);
11981 d7a4a5ae2dae Module unicodedata -- Provides access to the Unicode 3.0 data base. Guido van Rossum <guido@python.org> parents: diff changeset	1273 }
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1274
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59819 diff changeset	1275 /*
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1276 Local variables:
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1277 c-basic-offset: 4
26379 1885b2d6393a Patch #626485: Support Unicode normalization. Martin v. Löwis <martin@v.loewis.de> parents: 26376 diff changeset	1278 indent-tabs-mode: nil
26370 9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1279 End:
9e489a0df7b5 Patch #626548: Support Hangul syllable names. Martin v. Löwis <martin@v.loewis.de> parents: 25852 diff changeset	1280 */

Mercurial > cpython-lru_cache-weakref

annotate Modules/unicodedata.c @ 80661:94de77bd0b4b 2.7