Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_normalization.py @ 57114:d3a81051663b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
importlib.test.source.util referenced variables in the 'finally' part of a try/finally which may not have been set.
author Brett Cannon <bcannon@gmail.com>
date Thu, 05 Nov 2009 01:17:22 +0000
parents 3bf761df802a
children 74230e5ab373
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47201
9c930a32cebb #2621 rename test.test_support to test.support
Benjamin Peterson <benjamin@python.org>
parents: 43089
diff changeset
1 from test.support import run_unittest, open_urlresource
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
2 import unittest
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
3
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
4 import sys
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
5 import os
49133
f97779262bac Merged revisions 66369 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49130
diff changeset
6 from unicodedata import normalize, unidata_version
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
7
42622
4d2aa4bfb2d6 Remove RISCOS support
Skip Montanaro <skip@pobox.com>
parents: 41585
diff changeset
8 TESTDATAFILE = "NormalizationTest.txt"
49133
f97779262bac Merged revisions 66369 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49130
diff changeset
9 TESTDATAURL = "http://www.unicode.org/Public/" + unidata_version + "/ucd/" + TESTDATAFILE
f97779262bac Merged revisions 66369 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49130
diff changeset
10
55394
abfd5f78f466 Make test_normalization verify the version of the correct test data file.
Alexandre Vassalotti <alexandre@peadrop.com>
parents: 55593
diff changeset
11 # Verify we have the correct version of the test data file.
abfd5f78f466 Make test_normalization verify the version of the correct test data file.
Alexandre Vassalotti <alexandre@peadrop.com>
parents: 55593
diff changeset
12 TESTDATAPATH = os.path.join(os.path.dirname(__file__), "data", TESTDATAFILE)
abfd5f78f466 Make test_normalization verify the version of the correct test data file.
Alexandre Vassalotti <alexandre@peadrop.com>
parents: 55593
diff changeset
13 if os.path.exists(TESTDATAPATH):
abfd5f78f466 Make test_normalization verify the version of the correct test data file.
Alexandre Vassalotti <alexandre@peadrop.com>
parents: 55593
diff changeset
14 f = open(TESTDATAPATH, encoding='utf-8')
49133
f97779262bac Merged revisions 66369 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49130
diff changeset
15 l = f.readline()
f97779262bac Merged revisions 66369 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49130
diff changeset
16 f.close()
f97779262bac Merged revisions 66369 via svnmerge from
Martin v. Löwis <martin@v.loewis.de>
parents: 49130
diff changeset
17 if not unidata_version in l:
55394
abfd5f78f466 Make test_normalization verify the version of the correct test data file.
Alexandre Vassalotti <alexandre@peadrop.com>
parents: 55593
diff changeset
18 os.unlink(testdatafile)
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
19
37188
760ffb0fba37 Exceptions should inherit from Exception now.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36937
diff changeset
20 class RangeError(Exception):
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
21 pass
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
22
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
23 def NFC(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
24 return normalize("NFC", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
25
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
26 def NFKC(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
27 return normalize("NFKC", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
28
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
29 def NFD(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
30 return normalize("NFD", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
31
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
32 def NFKD(str):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
33 return normalize("NFKD", str)
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
34
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
35 def unistr(data):
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
36 data = [int(x, 16) for x in data.split(" ")]
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
37 for x in data:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
38 if x > sys.maxunicode:
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
39 raise RangeError
41585
825d966338ea Rename 'unicode' to 'str' in its tp_name field. Rename 'str' to 'str8'.
Guido van Rossum <guido@python.org>
parents: 41575
diff changeset
40 return "".join([chr(x) for x in data])
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
41
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
42 class NormalizationTest(unittest.TestCase):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
43 def test_main(self):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
44 part1_data = {}
57051
3bf761df802a Merged revisions 76033 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 55394
diff changeset
45 # Hit the exception early
3bf761df802a Merged revisions 76033 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 55394
diff changeset
46 try:
3bf761df802a Merged revisions 76033 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 55394
diff changeset
47 open_urlresource(TESTDATAURL, encoding="utf-8")
3bf761df802a Merged revisions 76033 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 55394
diff changeset
48 except IOError:
3bf761df802a Merged revisions 76033 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 55394
diff changeset
49 self.skipTest("Could not retrieve " + TESTDATAURL)
43089
2e3c02c625de Explicitly use UTF-8 as the encoding for the normalization file.
Martin v. Löwis <martin@v.loewis.de>
parents: 42622
diff changeset
50 for line in open_urlresource(TESTDATAURL, encoding="utf-8"):
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
51 if '#' in line:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
52 line = line.split('#')[0]
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
53 line = line.strip()
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
54 if not line:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
55 continue
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
56 if line.startswith("@Part"):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
57 part = line.split()[0]
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
58 continue
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
59 if part == "@Part3":
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
60 # XXX we don't support PRI #29 yet, so skip these tests for now
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
61 continue
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
62 try:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
63 c1,c2,c3,c4,c5 = [unistr(x) for x in line.split(';')[:-1]]
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
64 except RangeError:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
65 # Skip unsupported characters;
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
66 # try atleast adding c1 if we are in part1
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
67 if part == "@Part1":
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
68 try:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
69 c1 = unistr(line.split(';')[0])
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
70 except RangeError:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
71 pass
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
72 else:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
73 part1_data[c1] = 1
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
74 continue
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
75
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
76 # Perform tests
55593
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
77 self.assertTrue(c2 == NFC(c1) == NFC(c2) == NFC(c3), line)
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
78 self.assertTrue(c4 == NFC(c4) == NFC(c5), line)
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
79 self.assertTrue(c3 == NFD(c1) == NFD(c2) == NFD(c3), line)
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
80 self.assertTrue(c5 == NFD(c4) == NFD(c5), line)
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
81 self.assertTrue(c4 == NFKC(c1) == NFKC(c2) == \
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
82 NFKC(c3) == NFKC(c4) == NFKC(c5),
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
83 line)
55593
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
84 self.assertTrue(c5 == NFKD(c1) == NFKD(c2) == \
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
85 NFKD(c3) == NFKD(c4) == NFKD(c5),
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
86 line)
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
87
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
88 # Record part 1 data
36929
fab44c5c8df4 Update test data to 4.1; disable PRI #29 for now.
Martin v. Löwis <martin@v.loewis.de>
parents: 35998
diff changeset
89 if part == "@Part1":
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
90 part1_data[c1] = 1
26387
fcf9734e1021 Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 26379
diff changeset
91
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
92 # Perform tests for all other data
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
93 for c in range(sys.maxunicode+1):
41585
825d966338ea Rename 'unicode' to 'str' in its tp_name field. Rename 'str' to 'str8'.
Guido van Rossum <guido@python.org>
parents: 41575
diff changeset
94 X = chr(c)
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
95 if X in part1_data:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
96 continue
55593
0bb232078b8c Merged revisions 73715 via svnmerge from
Georg Brandl <georg@python.org>
parents: 50203
diff changeset
97 self.assertTrue(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c)
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
98
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
99 def test_bug_834676(self):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
100 # Check for bug 834676
41575
beacf40b5960 Rip out all the u"..." literals and calls to unicode().
Guido van Rossum <guido@python.org>
parents: 41554
diff changeset
101 normalize('NFC', '\ud55c\uae00')
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
102
26379
1885b2d6393a Patch #626485: Support Unicode normalization.
Martin v. Löwis <martin@v.loewis.de>
parents:
diff changeset
103
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
104 def test_main():
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
105 run_unittest(NormalizationTest)
30867
04ca4d1f751f Overallocate target buffer for normalization more early. Fixes #834676.
Martin v. Löwis <martin@v.loewis.de>
parents: 30056
diff changeset
106
26391
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
107 if __name__ == "__main__":
d207d1d78c5d Fiddled things so that test_normalization is expected to be skipped if
Tim Peters <tim.peters@gmail.com>
parents: 26388
diff changeset
108 test_main()