Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_normalization.py @ 58765:2ba7677bfa95 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
testCopysign was defined twice in test_math; combine the definitions
author Mark Dickinson <dickinsm@gmail.com>
date Sat, 06 Feb 2010 23:11:25 +0000
parents 7b74abb80356
children c4dd30b5d07e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
1 from test.test_support import run_unittest, open_urlresource
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
2 import unittest
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
3
57466
7b74abb80356 When open_urlresource() fails, HTTPException is another possible error
Antoine Pitrou <solipsis@pitrou.net>
parents: 57048
diff changeset
4 from httplib import HTTPException
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
5 import sys
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
6 import os
49131
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
7 from unicodedata import normalize, unidata_version
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
8
36929
fab44c5c8df4 Update test data to 4.1; disable PRI #29 for now.
Martin v. Löwis <martin@v.loewis.de>
parents: 35998
diff changeset
9 TESTDATAFILE = "NormalizationTest" + os.extsep + "txt"
49131
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
10 TESTDATAURL = "http://www.unicode.org/Public/" + unidata_version + "/ucd/" + TESTDATAFILE
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
11
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
12 if os.path.exists(TESTDATAFILE):
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
13 f = open(TESTDATAFILE)
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
14 l = f.readline()
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
15 f.close()
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
16 if not unidata_version in l:
2ab85c392a8d Read unidata_version from unicodedata module.
Martin v. Löwis <martin@v.loewis.de>
parents: 49129
diff changeset
17 os.unlink(TESTDATAFILE)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
18
37188
760ffb0fba37 Exceptions should inherit from Exception now.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36937
diff changeset
19 class RangeError(Exception):
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
20 pass
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
21
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
22 def NFC(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
23 return normalize("NFC", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
24
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
25 def NFKC(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
26 return normalize("NFKC", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
27
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
28 def NFD(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
29 return normalize("NFD", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
30
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
31 def NFKD(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
32 return normalize("NFKD", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
33
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
34 def unistr(data):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
35 data = [int(x, 16) for x in data.split(" ")]
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
36 for x in data:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
37 if x > sys.maxunicode:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
38 raise RangeError
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
39 return u"".join([unichr(x) for x in data])
41532
4b9fa8092e82 Whitespace normalization. Ugh, we really need to do this more often.
Neal Norwitz <nnorwitz@gmail.com>
parents: 41472
diff changeset
40
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
41 class NormalizationTest(unittest.TestCase):
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
42 def test_main(self):
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
43 part1_data = {}
57048
8ca4d77680c6 test_normalization should skip and not crash when the resource isn't available
Antoine Pitrou <solipsis@pitrou.net>
parents: 55002
diff changeset
44 # Hit the exception early
8ca4d77680c6 test_normalization should skip and not crash when the resource isn't available
Antoine Pitrou <solipsis@pitrou.net>
parents: 55002
diff changeset
45 try:
8ca4d77680c6 test_normalization should skip and not crash when the resource isn't available
Antoine Pitrou <solipsis@pitrou.net>
parents: 55002
diff changeset
46 open_urlresource(TESTDATAURL)
57466
7b74abb80356 When open_urlresource() fails, HTTPException is another possible error
Antoine Pitrou <solipsis@pitrou.net>
parents: 57048
diff changeset
47 except (IOError, HTTPException):
57048
8ca4d77680c6 test_normalization should skip and not crash when the resource isn't available
Antoine Pitrou <solipsis@pitrou.net>
parents: 55002
diff changeset
48 self.skipTest("Could not retrieve " + TESTDATAURL)
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
49 for line in open_urlresource(TESTDATAURL):
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
50 if '#' in line:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
51 line = line.split('#')[0]
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
52 line = line.strip()
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
53 if not line:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
54 continue
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
55 if line.startswith("@Part"):
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
56 part = line.split()[0]
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
57 continue
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
58 if part == "@Part3":
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
59 # XXX we don't support PRI #29 yet, so skip these tests for now
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
60 continue
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
61 try:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
62 c1,c2,c3,c4,c5 = [unistr(x) for x in line.split(';')[:-1]]
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
63 except RangeError:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
64 # Skip unsupported characters;
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
65 # try atleast adding c1 if we are in part1
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
66 if part == "@Part1":
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
67 try:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
68 c1 = unistr(line.split(';')[0])
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
69 except RangeError:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
70 pass
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
71 else:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
72 part1_data[c1] = 1
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
73 continue
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
74
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
75 # Perform tests
55002
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
76 self.assertTrue(c2 == NFC(c1) == NFC(c2) == NFC(c3), line)
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
77 self.assertTrue(c4 == NFC(c4) == NFC(c5), line)
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
78 self.assertTrue(c3 == NFD(c1) == NFD(c2) == NFD(c3), line)
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
79 self.assertTrue(c5 == NFD(c4) == NFD(c5), line)
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
80 self.assertTrue(c4 == NFKC(c1) == NFKC(c2) == \
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
81 NFKC(c3) == NFKC(c4) == NFKC(c5),
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
82 line)
55002
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
83 self.assertTrue(c5 == NFKD(c1) == NFKD(c2) == \
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
84 NFKD(c3) == NFKD(c4) == NFKD(c5),
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
85 line)
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
86
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
87 # Record part 1 data
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
88 if part == "@Part1":
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
89 part1_data[c1] = 1
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
90
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
91 # Perform tests for all other data
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
92 for c in range(sys.maxunicode+1):
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
93 X = unichr(c)
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
94 if X in part1_data:
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
95 continue
55002
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 49131
diff changeset
96 self.assertTrue(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c)
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
97
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
98 def test_bug_834676(self):
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
99 # Check for bug 834676
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
100 normalize('NFC', u'\ud55c\uae00')
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
101
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
102
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
103 def test_main():
41393
ea6fbf404c34 Convert test_normalization to use unittest.
Collin Winter <collinw@gmail.com>
parents: 37188
diff changeset
104 run_unittest(NormalizationTest)
30867
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 30056
diff changeset
105
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
106 if __name__ == "__main__":
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
107 test_main()