Mercurial > cpython-lru_cache-weakref

annotate Modules/cjkcodecs/README @ 78069:a8aa77458e12 3.2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add c_size_t and c_ssize_t to the ctypes summary table.
author Antoine Pitrou <solipsis@pitrou.net>
date Thu, 12 Jul 2012 20:31:50 +0200
parents bac6bd343a12
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
1 To generate or modify mapping headers
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
2 -------------------------------------
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
3 Mapping headers are imported from CJKCodecs as pre-generated form.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
4 If you need to tweak or add something on it, please look at tools/
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
5 subdirectory of CJKCodecs' distribution.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
6
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
7
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
8
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
9 Notes on implmentation characteristics of each codecs
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
10 -----------------------------------------------------
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
11
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
12 1) Big5 codec
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
13
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
14 The big5 codec maps the following characters as cp950 does rather
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
15 than conforming Unicode.org's that maps to 0xFFFD.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
16
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
17 BIG5 Unicode Description
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
18
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
19 0xA15A 0x2574 SPACING UNDERSCORE
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
20 0xA1C3 0xFFE3 SPACING HEAVY OVERSCORE
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
21 0xA1C5 0x02CD SPACING HEAVY UNDERSCORE
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
22 0xA1FE 0xFF0F LT DIAG UP RIGHT TO LOW LEFT
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
23 0xA240 0xFF3C LT DIAG UP LEFT TO LOW RIGHT
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
24 0xA2CC 0x5341 HANGZHOU NUMERAL TEN
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
25 0xA2CE 0x5345 HANGZHOU NUMERAL THIRTY
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
26
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
27 Because unicode 0x5341, 0x5345, 0xFF0F, 0xFF3C is mapped to another
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
28 big5 codes already, a roundtrip compatibility is not guaranteed for
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
29 them.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
30
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
31
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
32 2) cp932 codec
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
33
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
34 To conform to Windows's real mapping, cp932 codec maps the following
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
35 codepoints in addition of the official cp932 mapping.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
36
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
37 CP932 Unicode Description
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
38
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
39 0x80 0x80 UNDEFINED
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
40 0xA0 0xF8F0 UNDEFINED
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
41 0xFD 0xF8F1 UNDEFINED
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
42 0xFE 0xF8F2 UNDEFINED
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
43 0xFF 0xF8F3 UNDEFINED
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
44
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
45
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
46 3) euc-jisx0213 codec
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
47
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
48 The euc-jisx0213 codec maps JIS X 0213 Plane 1 code 0x2140 into
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
49 unicode U+FF3C instead of U+005C as on unicode.org's mapping.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
50 Because euc-jisx0213 has REVERSE SOLIDUS on 0x5c already and A140
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
51 is shown as a full width character, mapping to U+FF3C can make
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
52 more sense.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
53
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
54 The euc-jisx0213 codec is enabled to decode JIS X 0212 codes on
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
55 codeset 2. Because JIS X 0212 and JIS X 0213 Plane 2 don't have
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
56 overlapped by each other, it doesn't bother standard conformations
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
57 (and JIS X 0213 Plane 2 is intended to use so.) On encoding
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
58 sessions, the codec will try to encode kanji characters in this
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
59 order:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
60
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
61 JIS X 0213 Plane 1 -> JIS X 0213 Plane 2 -> JIS X 0212
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
62
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
63
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
64 4) euc-jp codec
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
65
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
66 The euc-jp codec is a compatibility instance on these points:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
67 - U+FF3C FULLWIDTH REVERSE SOLIDUS is mapped to EUC-JP A1C0 (vice versa)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
68 - U+00A5 YEN SIGN is mapped to EUC-JP 0x5c. (one way)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
69 - U+203E OVERLINE is mapped to EUC-JP 0x7e. (one way)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
70
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
71
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
72 5) shift-jis codec
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
73
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
74 The shift-jis codec is mapping 0x20-0x7e area to U+20-U+7E directly
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
75 instead of using JIS X 0201 for compatibility. The differences are:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
76 - U+005C REVERSE SOLIDUS is mapped to SHIFT-JIS 0x5c.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
77 - U+007E TILDE is mapped to SHIFT-JIS 0x7e.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
78 - U+FF3C FULL-WIDTH REVERSE SOLIDUS is mapped to SHIFT-JIS 815f.
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
79