Mercurial > cpython-lru_cache-weakref

annotate Objects/unicodectype.c @ 78069:a8aa77458e12 3.2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add c_size_t and c_ssize_t to the ctypes summary table.
author Antoine Pitrou <solipsis@pitrou.net>
date Thu, 12 Jul 2012 20:31:50 +0200
parents 3c9232a684ae
children f7e05d205a52
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 /*
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
2 Unicode character type helpers.
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
3
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
4 Written by Marc-Andre Lemburg (mal@lemburg.com).
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
5 Modified for Python 2.0 by Fredrik Lundh (fredrik@pythonware.com)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
7 Copyright (c) Corporation for National Research Initiatives.
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
9 */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 #include "Python.h"
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
12
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
13 #define ALPHA_MASK 0x01
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
14 #define DECIMAL_MASK 0x02
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
15 #define DIGIT_MASK 0x04
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
16 #define LOWER_MASK 0x08
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
17 #define LINEBREAK_MASK 0x10
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
18 #define SPACE_MASK 0x20
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
19 #define TITLE_MASK 0x40
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
20 #define UPPER_MASK 0x80
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
21 #define XID_START_MASK 0x100
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
22 #define XID_CONTINUE_MASK 0x200
48427
4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters.
Georg Brandl <georg@python.org>
parents: 47611
diff changeset
23 #define PRINTABLE_MASK 0x400
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
24 #define NODELTA_MASK 0x800
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53452
diff changeset
25 #define NUMERIC_MASK 0x1000
13380
d166a1e60760 Conditionally (currently on ifdef macintosh) break the large switch up
Jack Jansen <jack.jansen@cwi.nl>
parents: 13355
diff changeset
26
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
27 typedef struct {
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
28 const Py_UCS4 upper;
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
29 const Py_UCS4 lower;
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
30 const Py_UCS4 title;
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
31 const unsigned char decimal;
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
32 const unsigned char digit;
32197
3a84e23b042f - SF #962502: Add two more methods for unicode type; width() and
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31273
diff changeset
33 const unsigned short flags;
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
34 } _PyUnicode_TypeRecord;
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
35
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
36 #include "unicodetype_db.h"
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
37
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
38 static const _PyUnicode_TypeRecord *
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
39 gettyperecord(Py_UCS4 code)
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
40 {
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
41 int index;
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
42
25852
1f5f6481886d Update to Unicode 3.2 database.
Martin v. Löwis <martin@v.loewis.de>
parents: 18332
diff changeset
43 if (code >= 0x110000)
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
44 index = 0;
31273
25519d6c97b8 Fix gcc 3.3 warnings related to Py_UNICODE_WIDE.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 25856
diff changeset
45 else
25519d6c97b8 Fix gcc 3.3 warnings related to Py_UNICODE_WIDE.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 25856
diff changeset
46 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
47 index = index1[(code>>SHIFT)];
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
48 index = index2[(index<<SHIFT)+(code&((1<<SHIFT)-1))];
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
49 }
18287
f6078a30f70a more unicode tweaks: fix unicodectype for sizeof(Py_UNICODE) >
Fredrik Lundh <fredrik@pythonware.com>
parents: 14854
diff changeset
50
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
51 return &_PyUnicode_TypeRecords[index];
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
52 }
13380
d166a1e60760 Conditionally (currently on ifdef macintosh) break the large switch up
Jack Jansen <jack.jansen@cwi.nl>
parents: 13355
diff changeset
53
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 /* Returns the titlecase Unicode characters corresponding to ch or just
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 ch if no titlecase mapping is known. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
56
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
57 Py_UCS4 _PyUnicode_ToTitlecase(register Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
59 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
53452
912438655f93 Merged revisions 71949 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49125
diff changeset
60 int delta = ctype->title;
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
61
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
62 if (ctype->flags & NODELTA_MASK)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
63 return delta;
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
64
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
65 if (delta >= 32768)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
66 delta -= 65536;
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
67
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
68 return ch + delta;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
69 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
70
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
71 /* Returns 1 for Unicode characters having the category 'Lt', 0
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
72 otherwise. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
73
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
74 int _PyUnicode_IsTitlecase(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
75 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
76 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
77
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
78 return (ctype->flags & TITLE_MASK) != 0;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
79 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
80
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
81 /* Returns 1 for Unicode characters having the XID_Start property, 0
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
82 otherwise. */
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
83
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
84 int _PyUnicode_IsXidStart(Py_UCS4 ch)
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
85 {
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
86 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
87
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
88 return (ctype->flags & XID_START_MASK) != 0;
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
89 }
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
90
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
91 /* Returns 1 for Unicode characters having the XID_Continue property,
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
92 0 otherwise. */
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
93
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
94 int _PyUnicode_IsXidContinue(Py_UCS4 ch)
42595
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
95 {
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
96 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
97
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
98 return (ctype->flags & XID_CONTINUE_MASK) != 0;
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
99 }
f6af0b2598ca Add XID_Start and XID_Continue properties to unicodectype.
Martin v. Löwis <martin@v.loewis.de>
parents: 38453
diff changeset
100
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 /* Returns the integer decimal (0-9) for Unicode characters having
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
102 this property, -1 otherwise. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
103
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
104 int _PyUnicode_ToDecimalDigit(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
105 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
106 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
107
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
108 return (ctype->flags & DECIMAL_MASK) ? ctype->decimal : -1;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
109 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
110
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
111 int _PyUnicode_IsDecimalDigit(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
112 {
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
113 if (_PyUnicode_ToDecimalDigit(ch) < 0)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
114 return 0;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
115 return 1;
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
116 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
117
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
118 /* Returns the integer digit (0-9) for Unicode characters having
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 this property, -1 otherwise. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
120
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
121 int _PyUnicode_ToDigit(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
123 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
124
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
125 return (ctype->flags & DIGIT_MASK) ? ctype->digit : -1;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
127
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
128 int _PyUnicode_IsDigit(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 {
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
130 if (_PyUnicode_ToDigit(ch) < 0)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
131 return 0;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 return 1;
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
134
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 /* Returns the numeric value as double for Unicode characters having
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
136 this property, -1.0 otherwise. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
137
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
138 int _PyUnicode_IsNumeric(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
139 {
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53452
diff changeset
140 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
141
56408
6c91d1803182 Merged revisions 75272-75273 via svnmerge from
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 53452
diff changeset
142 return (ctype->flags & NUMERIC_MASK) != 0;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
143 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
144
47611
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
145 /* Returns 1 for Unicode characters to be hex-escaped when repr()ed,
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
146 0 otherwise.
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
147 All characters except those characters defined in the Unicode character
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
148 database as following categories are considered printable.
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
149 * Cc (Other, Control)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
150 * Cf (Other, Format)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
151 * Cs (Other, Surrogate)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
152 * Co (Other, Private Use)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
153 * Cn (Other, Not Assigned)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
154 * Zl Separator, Line ('\u2028', LINE SEPARATOR)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
155 * Zp Separator, Paragraph ('\u2029', PARAGRAPH SEPARATOR)
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
156 * Zs (Separator, Space) other than ASCII space('\x20').
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
157 */
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
158 int _PyUnicode_IsPrintable(Py_UCS4 ch)
47611
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
159 {
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
160 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
161
48427
4e6859c6d38c Issue #3282: str.isprintable() should return False for undefined Unicode characters.
Georg Brandl <georg@python.org>
parents: 47611
diff changeset
162 return (ctype->flags & PRINTABLE_MASK) != 0;
47611
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
163 }
da47e7e135ae #2630: Implement PEP 3138.
Georg Brandl <georg@python.org>
parents: 47463
diff changeset
164
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
165 /* Returns 1 for Unicode characters having the category 'Ll', 0
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
166 otherwise. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
167
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
168 int _PyUnicode_IsLowercase(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
169 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
170 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
171
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
172 return (ctype->flags & LOWER_MASK) != 0;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
174
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 /* Returns 1 for Unicode characters having the category 'Lu', 0
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
176 otherwise. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
177
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
178 int _PyUnicode_IsUppercase(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
179 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
180 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
181
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
182 return (ctype->flags & UPPER_MASK) != 0;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
184
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 /* Returns the uppercase Unicode characters corresponding to ch or just
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
186 ch if no uppercase mapping is known. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
187
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
188 Py_UCS4 _PyUnicode_ToUppercase(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
189 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
190 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
191 int delta = ctype->upper;
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
192 if (ctype->flags & NODELTA_MASK)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
193 return delta;
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
194 if (delta >= 32768)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
195 delta -= 65536;
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
196 return ch + delta;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
197 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
198
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
199 /* Returns the lowercase Unicode characters corresponding to ch or just
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
200 ch if no lowercase mapping is known. */
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
201
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
202 Py_UCS4 _PyUnicode_ToLowercase(Py_UCS4 ch)
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
203 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
204 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
205 int delta = ctype->lower;
49125
dc60293b29d5 Merged revisions 66362 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 48427
diff changeset
206 if (ctype->flags & NODELTA_MASK)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
207 return delta;
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
208 if (delta >= 32768)
61467
f197407bdba2 Recorded merge of revisions 81032 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 53452
diff changeset
209 delta -= 65536;
25856
84a3dd994cd1 Make lower/upper/title work for non-BMP characters.
Martin v. Löwis <martin@v.loewis.de>
parents: 25852
diff changeset
210 return ch + delta;
11967
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 }
507833b18213 Unicode character type helpers, written by Marc-Andre Lemburg.
Guido van Rossum <guido@python.org>
parents:
diff changeset
212
13355
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
213 /* Returns 1 for Unicode characters having the category 'Ll', 'Lu', 'Lt',
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
214 'Lo' or 'Lm', 0 otherwise. */
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
215
64146
8f97c64b2333 #5127: Even on narrow unicode builds, the C functions that access the Unicode
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 63924
diff changeset
216 int _PyUnicode_IsAlpha(Py_UCS4 ch)
13355
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
217 {
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
218 const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
13355
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
219
14854
89bc8475fc84 unicode database compression, step 3:
Fredrik Lundh <fredrik@pythonware.com>
parents: 13939
diff changeset
220 return (ctype->flags & ALPHA_MASK) != 0;
13355
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
221 }
b4aedd90c864 Added new lookup API which matches all alphabetic Unicode characters,
Marc-André Lemburg <mal@egenix.com>
parents: 12370
diff changeset
222