cpython-lru_cache-weakref: Objects/stringlib/codecs.h annotate

annotate Objects/stringlib/codecs.h @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	130597102dac
children

rev	line source
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	1 /* stringlib: codec implementations */
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	2
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	3 #if STRINGLIB_IS_UNICODE
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	4
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	5 /* Mask to quickly check whether a C 'long' contains a
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	6 non-ASCII, UTF8-encoded char. */
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	7 #if (SIZEOF_LONG == 8)
77966 ab9e824116eb Use correct types for ASCII_CHAR_MASK integer constants. Mark Dickinson <mdickinson@enthought.com> parents: 77649 diff changeset	8 # define ASCII_CHAR_MASK 0x8080808080808080UL
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	9 #elif (SIZEOF_LONG == 4)
77966 ab9e824116eb Use correct types for ASCII_CHAR_MASK integer constants. Mark Dickinson <mdickinson@enthought.com> parents: 77649 diff changeset	10 # define ASCII_CHAR_MASK 0x80808080UL
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	11 #else
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	12 # error C 'long' size should be either 4 or 8!
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	13 #endif
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	14
77649 3214c9ebcf5e Issue #14923: Optimize continuation-byte check in UTF-8 decoding. Patch by Serhiy Storchaka. Mark Dickinson <mdickinson@enthought.com> parents: 77448 diff changeset	15 /* 10xxxxxx */
3214c9ebcf5e Issue #14923: Optimize continuation-byte check in UTF-8 decoding. Patch by Serhiy Storchaka. Mark Dickinson <mdickinson@enthought.com> parents: 77448 diff changeset	16 #define IS_CONTINUATION_BYTE(ch) ((ch) >= 0x80 && (ch) < 0xC0)
3214c9ebcf5e Issue #14923: Optimize continuation-byte check in UTF-8 decoding. Patch by Serhiy Storchaka. Mark Dickinson <mdickinson@enthought.com> parents: 77448 diff changeset	17
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	18 Py_LOCAL_INLINE(Py_UCS4)
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	19 STRINGLIB(utf8_decode)(const char *inptr, const char end,
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	20 STRINGLIB_CHAR *dest,
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	21 Py_ssize_t *outpos)
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	22 {
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	23 Py_UCS4 ch;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	24 const char s = inptr;
79065 99112b851b25 Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t. Antoine Pitrou <solipsis@pitrou.net> parents: 77966 diff changeset	25 const char aligned_end = (const char ) _Py_ALIGN_DOWN(end, SIZEOF_LONG);
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	26 STRINGLIB_CHAR p = dest + outpos;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	27
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	28 while (s < end) {
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	29 ch = (unsigned char)*s;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	30
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	31 if (ch < 0x80) {
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	32 /* Fast path for runs of ASCII characters. Given that common UTF-8
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	33 input will consist of an overwhelming majority of ASCII
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	34 characters, we try to optimize for this case by checking
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	35 as many characters as a C 'long' can contain.
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	36 First, check if we can do an aligned read, as most CPUs have
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	37 a penalty for unaligned reads.
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	38 */
79065 99112b851b25 Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t. Antoine Pitrou <solipsis@pitrou.net> parents: 77966 diff changeset	39 if (_Py_IS_ALIGNED(s, SIZEOF_LONG)) {
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	40 /* Help register allocation */
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 80254 diff changeset	41 const char *_s = s;
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 80254 diff changeset	42 STRINGLIB_CHAR *_p = p;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	43 while (_s < aligned_end) {
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	44 /* Read a whole long at a time (either 4 or 8 bytes),
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	45 and do a fast unrolled copy if it only contains ASCII
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	46 characters. */
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	47 unsigned long value = (unsigned long ) _s;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	48 if (value & ASCII_CHAR_MASK)
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	49 break;
79803 d2127cdec10e Issue #16166: Add PY_LITTLE_ENDIAN and PY_BIG_ENDIAN macros and unified Christian Heimes <christian@cheimes.de> parents: 79065 diff changeset	50 #if PY_LITTLE_ENDIAN
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	51 _p[0] = (STRINGLIB_CHAR)(value & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	52 _p[1] = (STRINGLIB_CHAR)((value >> 8) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	53 _p[2] = (STRINGLIB_CHAR)((value >> 16) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	54 _p[3] = (STRINGLIB_CHAR)((value >> 24) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	55 # if SIZEOF_LONG == 8
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	56 _p[4] = (STRINGLIB_CHAR)((value >> 32) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	57 _p[5] = (STRINGLIB_CHAR)((value >> 40) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	58 _p[6] = (STRINGLIB_CHAR)((value >> 48) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	59 _p[7] = (STRINGLIB_CHAR)((value >> 56) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	60 # endif
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	61 #else
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	62 # if SIZEOF_LONG == 8
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	63 _p[0] = (STRINGLIB_CHAR)((value >> 56) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	64 _p[1] = (STRINGLIB_CHAR)((value >> 48) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	65 _p[2] = (STRINGLIB_CHAR)((value >> 40) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	66 _p[3] = (STRINGLIB_CHAR)((value >> 32) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	67 _p[4] = (STRINGLIB_CHAR)((value >> 24) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	68 _p[5] = (STRINGLIB_CHAR)((value >> 16) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	69 _p[6] = (STRINGLIB_CHAR)((value >> 8) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	70 _p[7] = (STRINGLIB_CHAR)(value & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	71 # else
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	72 _p[0] = (STRINGLIB_CHAR)((value >> 24) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	73 _p[1] = (STRINGLIB_CHAR)((value >> 16) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	74 _p[2] = (STRINGLIB_CHAR)((value >> 8) & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	75 _p[3] = (STRINGLIB_CHAR)(value & 0xFFu);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	76 # endif
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	77 #endif
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	78 _s += SIZEOF_LONG;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	79 _p += SIZEOF_LONG;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	80 }
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	81 s = _s;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	82 p = _p;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	83 if (s == end)
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	84 break;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	85 ch = (unsigned char)*s;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	86 }
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	87 if (ch < 0x80) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	88 s++;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	89 *p++ = ch;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	90 continue;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	91 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	92 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	93
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	94 if (ch < 0xE0) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	95 /* \xC2\x80-\xDF\xBF -- 0080-07FF */
80253 96f4cee8ea5e Issue #8271: Fix compilation on Windows Victor Stinner <victor.stinner@gmail.com> parents: 80246 diff changeset	96 Py_UCS4 ch2;
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	97 if (ch < 0xC2) {
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	98 /* invalid sequence
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	99 \x80-\xBF -- continuation byte
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	100 \xC0-\xC1 -- fake 0000-007F */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	101 goto InvalidStart;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	102 }
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	103 if (end - s < 2) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	104 /* unexpected end of data: the caller will decide whether
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	105 it's an error or not */
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	106 break;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	107 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	108 ch2 = (unsigned char)s[1];
77649 3214c9ebcf5e Issue #14923: Optimize continuation-byte check in UTF-8 decoding. Patch by Serhiy Storchaka. Mark Dickinson <mdickinson@enthought.com> parents: 77448 diff changeset	109 if (!IS_CONTINUATION_BYTE(ch2))
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	110 /* invalid continuation byte */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	111 goto InvalidContinuation1;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	112 ch = (ch << 6) + ch2 -
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	113 ((0xC0 << 6) + 0x80);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	114 assert ((ch > 0x007F) && (ch <= 0x07FF));
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	115 s += 2;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	116 if (STRINGLIB_MAX_CHAR <= 0x007F \|\|
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	117 (STRINGLIB_MAX_CHAR < 0x07FF && ch > STRINGLIB_MAX_CHAR))
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	118 /* Out-of-range */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	119 goto Return;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	120 *p++ = ch;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	121 continue;
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	122 }
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	123
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	124 if (ch < 0xF0) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	125 /* \xE0\xA0\x80-\xEF\xBF\xBF -- 0800-FFFF */
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	126 Py_UCS4 ch2, ch3;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	127 if (end - s < 3) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	128 /* unexpected end of data: the caller will decide whether
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	129 it's an error or not */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	130 if (end - s < 2)
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	131 break;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	132 ch2 = (unsigned char)s[1];
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	133 if (!IS_CONTINUATION_BYTE(ch2) \|\|
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	134 (ch2 < 0xA0 ? ch == 0xE0 : ch == 0xED))
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	135 /* for clarification see comments below */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	136 goto InvalidContinuation1;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	137 break;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	138 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	139 ch2 = (unsigned char)s[1];
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	140 ch3 = (unsigned char)s[2];
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	141 if (!IS_CONTINUATION_BYTE(ch2)) {
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	142 /* invalid continuation byte */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	143 goto InvalidContinuation1;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	144 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	145 if (ch == 0xE0) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	146 if (ch2 < 0xA0)
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	147 /* invalid sequence
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	148 \xE0\x80\x80-\xE0\x9F\xBF -- fake 0000-0800 */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	149 goto InvalidContinuation1;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	150 } else if (ch == 0xED && ch2 >= 0xA0) {
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	151 /* Decoding UTF-8 sequences in range \xED\xA0\x80-\xED\xBF\xBF
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	152 will result in surrogates in range D800-DFFF. Surrogates are
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	153 not valid UTF-8 so they are rejected.
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	154 See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	155 (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	156 goto InvalidContinuation1;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	157 }
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	158 if (!IS_CONTINUATION_BYTE(ch3)) {
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	159 /* invalid continuation byte */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	160 goto InvalidContinuation2;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	161 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	162 ch = (ch << 12) + (ch2 << 6) + ch3 -
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	163 ((0xE0 << 12) + (0x80 << 6) + 0x80);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	164 assert ((ch > 0x07FF) && (ch <= 0xFFFF));
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	165 s += 3;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	166 if (STRINGLIB_MAX_CHAR <= 0x07FF \|\|
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	167 (STRINGLIB_MAX_CHAR < 0xFFFF && ch > STRINGLIB_MAX_CHAR))
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	168 /* Out-of-range */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	169 goto Return;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	170 *p++ = ch;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	171 continue;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	172 }
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	173
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	174 if (ch < 0xF5) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	175 /* \xF0\x90\x80\x80-\xF4\x8F\xBF\xBF -- 10000-10FFFF */
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	176 Py_UCS4 ch2, ch3, ch4;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	177 if (end - s < 4) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	178 /* unexpected end of data: the caller will decide whether
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	179 it's an error or not */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	180 if (end - s < 2)
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	181 break;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	182 ch2 = (unsigned char)s[1];
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	183 if (!IS_CONTINUATION_BYTE(ch2) \|\|
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	184 (ch2 < 0x90 ? ch == 0xF0 : ch == 0xF4))
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	185 /* for clarification see comments below */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	186 goto InvalidContinuation1;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	187 if (end - s < 3)
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	188 break;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	189 ch3 = (unsigned char)s[2];
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	190 if (!IS_CONTINUATION_BYTE(ch3))
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	191 goto InvalidContinuation2;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	192 break;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	193 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	194 ch2 = (unsigned char)s[1];
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	195 ch3 = (unsigned char)s[2];
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	196 ch4 = (unsigned char)s[3];
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	197 if (!IS_CONTINUATION_BYTE(ch2)) {
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	198 /* invalid continuation byte */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	199 goto InvalidContinuation1;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	200 }
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	201 if (ch == 0xF0) {
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	202 if (ch2 < 0x90)
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	203 /* invalid sequence
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	204 \xF0\x80\x80\x80-\xF0\x8F\xBF\xBF -- fake 0000-FFFF */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	205 goto InvalidContinuation1;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	206 } else if (ch == 0xF4 && ch2 >= 0x90) {
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	207 /* invalid sequence
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	208 \xF4\x90\x80\80- -- 110000- overflow */
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	209 goto InvalidContinuation1;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	210 }
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	211 if (!IS_CONTINUATION_BYTE(ch3)) {
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	212 /* invalid continuation byte */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	213 goto InvalidContinuation2;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	214 }
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	215 if (!IS_CONTINUATION_BYTE(ch4)) {
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	216 /* invalid continuation byte */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	217 goto InvalidContinuation3;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	218 }
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	219 ch = (ch << 18) + (ch2 << 12) + (ch3 << 6) + ch4 -
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	220 ((0xF0 << 18) + (0x80 << 12) + (0x80 << 6) + 0x80);
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	221 assert ((ch > 0xFFFF) && (ch <= 0x10FFFF));
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	222 s += 4;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	223 if (STRINGLIB_MAX_CHAR <= 0xFFFF \|\|
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	224 (STRINGLIB_MAX_CHAR < 0x10FFFF && ch > STRINGLIB_MAX_CHAR))
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	225 /* Out-of-range */
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	226 goto Return;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	227 *p++ = ch;
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	228 continue;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	229 }
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	230 goto InvalidStart;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	231 }
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	232 ch = 0;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	233 Return:
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	234 *inptr = s;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	235 *outpos = p - dest;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	236 return ch;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	237 InvalidStart:
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	238 ch = 1;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	239 goto Return;
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	240 InvalidContinuation1:
76862 e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	241 ch = 2;
e08c3791f035 Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. Antoine Pitrou <solipsis@pitrou.net> parents: 74032 diff changeset	242 goto Return;
80246 5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	243 InvalidContinuation2:
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	244 ch = 3;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	245 goto Return;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	246 InvalidContinuation3:
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	247 ch = 4;
5962f192a483 #8271: the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti. Ezio Melotti <ezio.melotti@gmail.com> parents: 79065 diff changeset	248 goto Return;
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	249 }
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	250
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	251 #undef ASCII_CHAR_MASK
8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	252
74032 fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	253
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	254 /* UTF-8 encoder specialized for a Unicode kind to avoid the slow
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	255 PyUnicode_READ() macro. Delete some parts of the code depending on the kind:
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	256 UCS-1 strings don't need to handle surrogates for example. */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	257 Py_LOCAL_INLINE(PyObject *)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	258 STRINGLIB(utf8_encoder)(PyObject *unicode,
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	259 STRINGLIB_CHAR *data,
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	260 Py_ssize_t size,
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	261 const char *errors)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	262 {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	263 #define MAX_SHORT_UNICHARS 300 /* largest size we'll do on the stack */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	264
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	265 Py_ssize_t i; /* index into s of next input byte */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	266 PyObject result; / result string object */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	267 char p; / next free byte in output buffer */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	268 Py_ssize_t nallocated; /* number of result bytes allocated */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	269 Py_ssize_t nneeded; /* number of result bytes needed */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	270 #if STRINGLIB_SIZEOF_CHAR > 1
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	271 PyObject *errorHandler = NULL;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	272 PyObject *exc = NULL;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	273 PyObject *rep = NULL;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	274 #endif
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	275 #if STRINGLIB_SIZEOF_CHAR == 1
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	276 const Py_ssize_t max_char_size = 2;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	277 char stackbuf[MAX_SHORT_UNICHARS * 2];
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	278 #elif STRINGLIB_SIZEOF_CHAR == 2
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	279 const Py_ssize_t max_char_size = 3;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	280 char stackbuf[MAX_SHORT_UNICHARS * 3];
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	281 #else /* STRINGLIB_SIZEOF_CHAR == 4 */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	282 const Py_ssize_t max_char_size = 4;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	283 char stackbuf[MAX_SHORT_UNICHARS * 4];
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	284 #endif
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	285
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	286 assert(size >= 0);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	287
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	288 if (size <= MAX_SHORT_UNICHARS) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	289 /* Write into the stack buffer; nallocated can't overflow.
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	290 * At the end, we'll allocate exactly as much heap space as it
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	291 * turns out we need.
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	292 */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	293 nallocated = Py_SAFE_DOWNCAST(sizeof(stackbuf), size_t, int);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	294 result = NULL; /* will allocate after we're done */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	295 p = stackbuf;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	296 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	297 else {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	298 if (size > PY_SSIZE_T_MAX / max_char_size) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	299 /* integer overflow */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	300 return PyErr_NoMemory();
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	301 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	302 /* Overallocate on the heap, and give the excess back at the end. */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	303 nallocated = size * max_char_size;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	304 result = PyBytes_FromStringAndSize(NULL, nallocated);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	305 if (result == NULL)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	306 return NULL;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	307 p = PyBytes_AS_STRING(result);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	308 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	309
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	310 for (i = 0; i < size;) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	311 Py_UCS4 ch = data[i++];
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	312
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	313 if (ch < 0x80) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	314 /* Encode ASCII */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	315 *p++ = (char) ch;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	316
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	317 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	318 else
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	319 #if STRINGLIB_SIZEOF_CHAR > 1
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	320 if (ch < 0x0800)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	321 #endif
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	322 {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	323 /* Encode Latin-1 */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	324 *p++ = (char)(0xc0 \| (ch >> 6));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	325 *p++ = (char)(0x80 \| (ch & 0x3f));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	326 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	327 #if STRINGLIB_SIZEOF_CHAR > 1
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	328 else if (Py_UNICODE_IS_SURROGATE(ch)) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	329 Py_ssize_t newpos;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	330 Py_ssize_t repsize, k, startpos;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	331 startpos = i-1;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	332 rep = unicode_encode_call_errorhandler(
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	333 errors, &errorHandler, "utf-8", "surrogates not allowed",
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	334 unicode, &exc, startpos, startpos+1, &newpos);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	335 if (!rep)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	336 goto error;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	337
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	338 if (PyBytes_Check(rep))
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	339 repsize = PyBytes_GET_SIZE(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	340 else
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	341 repsize = PyUnicode_GET_LENGTH(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	342
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	343 if (repsize > max_char_size) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	344 Py_ssize_t offset;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	345
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	346 if (result == NULL)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	347 offset = p - stackbuf;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	348 else
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	349 offset = p - PyBytes_AS_STRING(result);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	350
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	351 if (nallocated > PY_SSIZE_T_MAX - repsize + max_char_size) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	352 /* integer overflow */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	353 PyErr_NoMemory();
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	354 goto error;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	355 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	356 nallocated += repsize - max_char_size;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	357 if (result != NULL) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	358 if (_PyBytes_Resize(&result, nallocated) < 0)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	359 goto error;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	360 } else {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	361 result = PyBytes_FromStringAndSize(NULL, nallocated);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	362 if (result == NULL)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	363 goto error;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	364 Py_MEMCPY(PyBytes_AS_STRING(result), stackbuf, offset);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	365 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	366 p = PyBytes_AS_STRING(result) + offset;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	367 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	368
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	369 if (PyBytes_Check(rep)) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	370 char *prep = PyBytes_AS_STRING(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	371 for(k = repsize; k > 0; k--)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	372 p++ = prep++;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	373 } else /* rep is unicode */ {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	374 enum PyUnicode_Kind repkind;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	375 void *repdata;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	376
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	377 if (PyUnicode_READY(rep) < 0)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	378 goto error;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	379 repkind = PyUnicode_KIND(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	380 repdata = PyUnicode_DATA(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	381
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	382 for(k=0; k<repsize; k++) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	383 Py_UCS4 c = PyUnicode_READ(repkind, repdata, k);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	384 if (0x80 <= c) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	385 raise_encode_exception(&exc, "utf-8",
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	386 unicode,
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	387 i-1, i,
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	388 "surrogates not allowed");
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	389 goto error;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	390 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	391 *p++ = (char)c;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	392 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	393 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	394 Py_CLEAR(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	395 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	396 else
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	397 #if STRINGLIB_SIZEOF_CHAR > 2
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	398 if (ch < 0x10000)
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	399 #endif
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	400 {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	401 *p++ = (char)(0xe0 \| (ch >> 12));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	402 *p++ = (char)(0x80 \| ((ch >> 6) & 0x3f));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	403 *p++ = (char)(0x80 \| (ch & 0x3f));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	404 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	405 #if STRINGLIB_SIZEOF_CHAR > 2
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	406 else /* ch >= 0x10000 */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	407 {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	408 assert(ch <= MAX_UNICODE);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	409 /* Encode UCS4 Unicode ordinals */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	410 *p++ = (char)(0xf0 \| (ch >> 18));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	411 *p++ = (char)(0x80 \| ((ch >> 12) & 0x3f));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	412 *p++ = (char)(0x80 \| ((ch >> 6) & 0x3f));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	413 *p++ = (char)(0x80 \| (ch & 0x3f));
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	414 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	415 #endif /* STRINGLIB_SIZEOF_CHAR > 2 */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	416 #endif /* STRINGLIB_SIZEOF_CHAR > 1 */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	417 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	418
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	419 if (result == NULL) {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	420 /* This was stack allocated. */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	421 nneeded = p - stackbuf;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	422 assert(nneeded <= nallocated);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	423 result = PyBytes_FromStringAndSize(stackbuf, nneeded);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	424 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	425 else {
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	426 /* Cut back to size actually needed. */
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	427 nneeded = p - PyBytes_AS_STRING(result);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	428 assert(nneeded <= nallocated);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	429 _PyBytes_Resize(&result, nneeded);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	430 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	431
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	432 #if STRINGLIB_SIZEOF_CHAR > 1
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	433 Py_XDECREF(errorHandler);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	434 Py_XDECREF(exc);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	435 #endif
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	436 return result;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	437
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	438 #if STRINGLIB_SIZEOF_CHAR > 1
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	439 error:
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	440 Py_XDECREF(rep);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	441 Py_XDECREF(errorHandler);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	442 Py_XDECREF(exc);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	443 Py_XDECREF(result);
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	444 return NULL;
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	445 #endif
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	446
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	447 #undef MAX_SHORT_UNICHARS
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	448 }
fbd797fc3809 Issue #13624: Write a specialized UTF-8 encoder to allow more optimization Victor Stinner <victor.stinner@haypocalc.com> parents: 73668 diff changeset	449
76971 cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	450 /* The pattern for constructing UCS2-repeated masks. */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	451 #if SIZEOF_LONG == 8
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	452 # define UCS2_REPEAT_MASK 0x0001000100010001ul
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	453 #elif SIZEOF_LONG == 4
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	454 # define UCS2_REPEAT_MASK 0x00010001ul
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	455 #else
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	456 # error C 'long' size should be either 4 or 8!
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	457 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	458
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	459 /* The mask for fast checking. */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	460 #if STRINGLIB_SIZEOF_CHAR == 1
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	461 /* The mask for fast checking of whether a C 'long' contains a
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	462 non-ASCII or non-Latin1 UTF16-encoded characters. */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	463 # define FAST_CHAR_MASK (UCS2_REPEAT_MASK * (0xFFFFu & ~STRINGLIB_MAX_CHAR))
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	464 #else
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	465 /* The mask for fast checking of whether a C 'long' may contain
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	466 UTF16-encoded surrogate characters. This is an efficient heuristic,
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	467 assuming that non-surrogate characters with a code point >= 0x8000 are
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	468 rare in most input.
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	469 */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	470 # define FAST_CHAR_MASK (UCS2_REPEAT_MASK * 0x8000u)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	471 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	472 /* The mask for fast byte-swapping. */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	473 #define STRIPPED_MASK (UCS2_REPEAT_MASK * 0x00FFu)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	474 /* Swap bytes. */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	475 #define SWAB(value) ((((value) >> 8) & STRIPPED_MASK) \| \
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	476 (((value) & STRIPPED_MASK) << 8))
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	477
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	478 Py_LOCAL_INLINE(Py_UCS4)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	479 STRINGLIB(utf16_decode)(const unsigned char *inptr, const unsigned char e,
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	480 STRINGLIB_CHAR dest, Py_ssize_t outpos,
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	481 int native_ordering)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	482 {
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	483 Py_UCS4 ch;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	484 const unsigned char *aligned_end =
79065 99112b851b25 Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t. Antoine Pitrou <solipsis@pitrou.net> parents: 77966 diff changeset	485 (const unsigned char *) _Py_ALIGN_DOWN(e, SIZEOF_LONG);
76971 cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	486 const unsigned char q = inptr;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	487 STRINGLIB_CHAR p = dest + outpos;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	488 /* Offsets from q for retrieving byte pairs in the right order. */
79803 d2127cdec10e Issue #16166: Add PY_LITTLE_ENDIAN and PY_BIG_ENDIAN macros and unified Christian Heimes <christian@cheimes.de> parents: 79065 diff changeset	489 #if PY_LITTLE_ENDIAN
76971 cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	490 int ihi = !!native_ordering, ilo = !native_ordering;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	491 #else
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	492 int ihi = !native_ordering, ilo = !!native_ordering;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	493 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	494 --e;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	495
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	496 while (q < e) {
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	497 Py_UCS4 ch2;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	498 /* First check for possible aligned read of a C 'long'. Unaligned
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	499 reads are more expensive, better to defer to another iteration. */
79065 99112b851b25 Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t. Antoine Pitrou <solipsis@pitrou.net> parents: 77966 diff changeset	500 if (_Py_IS_ALIGNED(q, SIZEOF_LONG)) {
76971 cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	501 /* Fast path for runs of in-range non-surrogate chars. */
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 80254 diff changeset	502 const unsigned char *_q = q;
76971 cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	503 while (_q < aligned_end) {
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	504 unsigned long block = * (unsigned long *) _q;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	505 if (native_ordering) {
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	506 /* Can use buffer directly */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	507 if (block & FAST_CHAR_MASK)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	508 break;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	509 }
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	510 else {
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	511 /* Need to byte-swap */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	512 if (block & SWAB(FAST_CHAR_MASK))
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	513 break;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	514 #if STRINGLIB_SIZEOF_CHAR == 1
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	515 block >>= 8;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	516 #else
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	517 block = SWAB(block);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	518 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	519 }
79803 d2127cdec10e Issue #16166: Add PY_LITTLE_ENDIAN and PY_BIG_ENDIAN macros and unified Christian Heimes <christian@cheimes.de> parents: 79065 diff changeset	520 #if PY_LITTLE_ENDIAN
76971 cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	521 # if SIZEOF_LONG == 4
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	522 p[0] = (STRINGLIB_CHAR)(block & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	523 p[1] = (STRINGLIB_CHAR)(block >> 16);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	524 # elif SIZEOF_LONG == 8
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	525 p[0] = (STRINGLIB_CHAR)(block & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	526 p[1] = (STRINGLIB_CHAR)((block >> 16) & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	527 p[2] = (STRINGLIB_CHAR)((block >> 32) & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	528 p[3] = (STRINGLIB_CHAR)(block >> 48);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	529 # endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	530 #else
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	531 # if SIZEOF_LONG == 4
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	532 p[0] = (STRINGLIB_CHAR)(block >> 16);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	533 p[1] = (STRINGLIB_CHAR)(block & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	534 # elif SIZEOF_LONG == 8
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	535 p[0] = (STRINGLIB_CHAR)(block >> 48);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	536 p[1] = (STRINGLIB_CHAR)((block >> 32) & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	537 p[2] = (STRINGLIB_CHAR)((block >> 16) & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	538 p[3] = (STRINGLIB_CHAR)(block & 0xFFFFu);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	539 # endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	540 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	541 _q += SIZEOF_LONG;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	542 p += SIZEOF_LONG / 2;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	543 }
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	544 q = _q;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	545 if (q >= e)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	546 break;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	547 }
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	548
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	549 ch = (q[ihi] << 8) \| q[ilo];
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	550 q += 2;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	551 if (!Py_UNICODE_IS_SURROGATE(ch)) {
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	552 #if STRINGLIB_SIZEOF_CHAR < 2
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	553 if (ch > STRINGLIB_MAX_CHAR)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	554 /* Out-of-range */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	555 goto Return;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	556 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	557 *p++ = (STRINGLIB_CHAR)ch;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	558 continue;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	559 }
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	560
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	561 /* UTF-16 code pair: */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	562 if (q >= e)
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	563 goto UnexpectedEnd;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	564 if (!Py_UNICODE_IS_HIGH_SURROGATE(ch))
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	565 goto IllegalEncoding;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	566 ch2 = (q[ihi] << 8) \| q[ilo];
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	567 q += 2;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	568 if (!Py_UNICODE_IS_LOW_SURROGATE(ch2))
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	569 goto IllegalSurrogate;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	570 ch = Py_UNICODE_JOIN_SURROGATES(ch, ch2);
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	571 #if STRINGLIB_SIZEOF_CHAR < 4
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	572 /* Out-of-range */
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	573 goto Return;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	574 #else
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	575 *p++ = (STRINGLIB_CHAR)ch;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	576 #endif
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	577 }
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	578 ch = 0;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	579 Return:
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	580 *inptr = q;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	581 *outpos = p - dest;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	582 return ch;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	583 UnexpectedEnd:
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	584 ch = 1;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	585 goto Return;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	586 IllegalEncoding:
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	587 ch = 2;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	588 goto Return;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	589 IllegalSurrogate:
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	590 ch = 3;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	591 goto Return;
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	592 }
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	593 #undef UCS2_REPEAT_MASK
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	594 #undef FAST_CHAR_MASK
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	595 #undef STRIPPED_MASK
cdcc816dea85 Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Antoine Pitrou <solipsis@pitrou.net> parents: 76862 diff changeset	596 #undef SWAB
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	597
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	598
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	599 #if STRINGLIB_MAX_CHAR >= 0x80
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	600 Py_LOCAL_INLINE(Py_ssize_t)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	601 STRINGLIB(utf16_encode)(const STRINGLIB_CHAR *in,
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	602 Py_ssize_t len,
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	603 unsigned short **outptr,
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	604 int native_ordering)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	605 {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	606 unsigned short out = outptr;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	607 const STRINGLIB_CHAR *end = in + len;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	608 #if STRINGLIB_SIZEOF_CHAR == 1
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	609 if (native_ordering) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	610 const STRINGLIB_CHAR *unrolled_end = in + _Py_SIZE_ROUND_DOWN(len, 4);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	611 while (in < unrolled_end) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	612 out[0] = in[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	613 out[1] = in[1];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	614 out[2] = in[2];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	615 out[3] = in[3];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	616 in += 4; out += 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	617 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	618 while (in < end) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	619 out++ = in++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	620 }
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	621 } else {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	622 # define SWAB2(CH) ((CH) << 8) /* high byte is zero */
79065 99112b851b25 Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t. Antoine Pitrou <solipsis@pitrou.net> parents: 77966 diff changeset	623 const STRINGLIB_CHAR *unrolled_end = in + _Py_SIZE_ROUND_DOWN(len, 4);
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	624 while (in < unrolled_end) {
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	625 out[0] = SWAB2(in[0]);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	626 out[1] = SWAB2(in[1]);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	627 out[2] = SWAB2(in[2]);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	628 out[3] = SWAB2(in[3]);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	629 in += 4; out += 4;
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	630 }
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	631 while (in < end) {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	632 Py_UCS4 ch = *in++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	633 *out++ = SWAB2((Py_UCS2)ch);
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	634 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	635 #undef SWAB2
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	636 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	637 *outptr = out;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	638 return len;
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	639 #else
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	640 if (native_ordering) {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	641 #if STRINGLIB_MAX_CHAR < 0x10000
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	642 const STRINGLIB_CHAR *unrolled_end = in + _Py_SIZE_ROUND_DOWN(len, 4);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	643 while (in < unrolled_end) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	644 /* check if any character is a surrogate character */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	645 if (((in[0] ^ 0xd800) &
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	646 (in[1] ^ 0xd800) &
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	647 (in[2] ^ 0xd800) &
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	648 (in[3] ^ 0xd800) & 0xf800) == 0)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	649 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	650 out[0] = in[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	651 out[1] = in[1];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	652 out[2] = in[2];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	653 out[3] = in[3];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	654 in += 4; out += 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	655 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	656 #endif
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	657 while (in < end) {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	658 Py_UCS4 ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	659 ch = *in++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	660 if (ch < 0xd800)
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	661 *out++ = ch;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	662 else if (ch < 0xe000)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	663 /* reject surrogate characters (U+DC800-U+DFFF) */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	664 goto fail;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	665 #if STRINGLIB_MAX_CHAR >= 0x10000
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	666 else if (ch >= 0x10000) {
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	667 out[0] = Py_UNICODE_HIGH_SURROGATE(ch);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	668 out[1] = Py_UNICODE_LOW_SURROGATE(ch);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	669 out += 2;
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	670 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	671 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	672 else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	673 *out++ = ch;
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	674 }
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	675 } else {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	676 #define SWAB2(CH) (((CH) << 8) \| ((CH) >> 8))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	677 #if STRINGLIB_MAX_CHAR < 0x10000
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	678 const STRINGLIB_CHAR *unrolled_end = in + _Py_SIZE_ROUND_DOWN(len, 4);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	679 while (in < unrolled_end) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	680 /* check if any character is a surrogate character */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	681 if (((in[0] ^ 0xd800) &
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	682 (in[1] ^ 0xd800) &
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	683 (in[2] ^ 0xd800) &
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	684 (in[3] ^ 0xd800) & 0xf800) == 0)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	685 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	686 out[0] = SWAB2(in[0]);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	687 out[1] = SWAB2(in[1]);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	688 out[2] = SWAB2(in[2]);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	689 out[3] = SWAB2(in[3]);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	690 in += 4; out += 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	691 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	692 #endif
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	693 while (in < end) {
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	694 Py_UCS4 ch = *in++;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	695 if (ch < 0xd800)
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	696 *out++ = SWAB2((Py_UCS2)ch);
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	697 else if (ch < 0xe000)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	698 /* reject surrogate characters (U+DC800-U+DFFF) */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	699 goto fail;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	700 #if STRINGLIB_MAX_CHAR >= 0x10000
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	701 else if (ch >= 0x10000) {
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	702 Py_UCS2 ch1 = Py_UNICODE_HIGH_SURROGATE(ch);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	703 Py_UCS2 ch2 = Py_UNICODE_LOW_SURROGATE(ch);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	704 out[0] = SWAB2(ch1);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	705 out[1] = SWAB2(ch2);
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	706 out += 2;
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	707 }
acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	708 #endif
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	709 else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	710 *out++ = SWAB2((Py_UCS2)ch);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	711 }
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	712 #undef SWAB2
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	713 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	714 *outptr = out;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	715 return len;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	716 fail:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	717 *outptr = out;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	718 return len - (end - in + 1);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	719 #endif
77448 acca141fda80 Issue #15026: utf-16 encoding is now significantly faster (up to 10x). Antoine Pitrou <solipsis@pitrou.net> parents: 76971 diff changeset	720 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	721 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	722
73668 8e6c4acaf530 Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case. Antoine Pitrou <solipsis@pitrou.net> parents: diff changeset	723 #endif /* STRINGLIB_IS_UNICODE */

Mercurial > cpython-lru_cache-weakref

annotate Objects/stringlib/codecs.h @ 87690:66c1c9f32567 default tip