cpython-lru_cache-weakref: Modules/cjkcodecs/multibytecodec.c annotate

annotate Modules/cjkcodecs/multibytecodec.c @ 79675:c0a2294de787 2.7

Issue #16199: Backport 2to3 typo fix from 3.2.

author	Chris Jerdonek <chris.jerdonek@gmail.com>
date	Thu, 11 Oct 2012 16:15:46 -0700
parents	e789b4cda872
children

rev	line source
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1 /*
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	2 * multibytecodec.c: Common Multibyte Codec Implementation
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	3 *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	4 * Written by Hye-Shik Chang <perky@FreeBSD.org>
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	5 */
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	6
36831 bac6bd343a12 - Modernize code to use Py_ssize_t more intensively. Hye-Shik Chang <hyeshik@gmail.com> parents: 36684 diff changeset	7 #define PY_SSIZE_T_CLEAN
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	8 #include "Python.h"
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	9 #include "structmember.h"
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	10 #include "multibytecodec.h"
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	11
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	12 typedef struct {
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	13 const Py_UNICODE inbuf, inbuf_top, *inbuf_end;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	14 unsigned char outbuf, outbuf_end;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	15 PyObject excobj, outobj;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	16 } MultibyteEncodeBuffer;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	17
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	18 typedef struct {
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	19 const unsigned char inbuf, inbuf_top, *inbuf_end;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	20 Py_UNICODE outbuf, outbuf_end;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	21 PyObject excobj, outobj;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	22 } MultibyteDecodeBuffer;
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	23
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	24 PyDoc_STRVAR(MultibyteCodec_Encode__doc__,
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	25 "I.encode(unicode[, errors]) -> (string, length consumed)\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	26 \n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	27 Return an encoded string version of `unicode'. errors may be given to\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	28 set a different error handling scheme. Default is 'strict' meaning that\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	29 encoding errors raise a UnicodeEncodeError. Other possible values are\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	30 'ignore', 'replace' and 'xmlcharrefreplace' as well as any other name\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	31 registered with codecs.register_error that can handle UnicodeEncodeErrors.");
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	32
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	33 PyDoc_STRVAR(MultibyteCodec_Decode__doc__,
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	34 "I.decode(string[, errors]) -> (unicodeobject, length consumed)\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	35 \n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	36 Decodes `string' using I, an MultibyteCodec instance. errors may be given\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	37 to set a different error handling scheme. Default is 'strict' meaning\n\
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	38 that encoding errors raise a UnicodeDecodeError. Other possible values\n\
49431 2e794757bdc8 Merged revisions 66766-66767,66771-66772,66774,66776,66783-66787,66790,66793,66797 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48266 diff changeset	39 are 'ignore' and 'replace' as well as any other name registered with\n\
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	40 codecs.register_error that is able to handle UnicodeDecodeErrors.");
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	41
36684 0bebeb393045 More unconsting. Martin v. Löwis <martin@v.loewis.de> parents: 36501 diff changeset	42 static char *codeckwarglist[] = {"input", "errors", NULL};
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	43 static char *incnewkwarglist[] = {"errors", NULL};
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	44 static char *incrementalkwarglist[] = {"input", "final", NULL};
36684 0bebeb393045 More unconsting. Martin v. Löwis <martin@v.loewis.de> parents: 36501 diff changeset	45 static char *streamkwarglist[] = {"stream", "errors", NULL};
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	46
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	47 static PyObject multibytecodec_encode(MultibyteCodec ,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	48 MultibyteCodec_State , const Py_UNICODE *, Py_ssize_t,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	49 PyObject *, int);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	50
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	51 #define MBENC_RESET MBENC_MAX<<1 /* reset after an encoding session */
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	52
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	53 static PyObject *
36831 bac6bd343a12 - Modernize code to use Py_ssize_t more intensively. Hye-Shik Chang <hyeshik@gmail.com> parents: 36684 diff changeset	54 make_tuple(PyObject *object, Py_ssize_t len)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	55 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	56 PyObject v, w;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	57
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	58 if (object == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	59 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	60
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	61 v = PyTuple_New(2);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	62 if (v == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	63 Py_DECREF(object);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	64 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	65 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	66 PyTuple_SET_ITEM(v, 0, object);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	67
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	68 w = PyInt_FromSsize_t(len);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	69 if (w == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	70 Py_DECREF(v);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	71 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	72 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	73 PyTuple_SET_ITEM(v, 1, w);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	74
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	75 return v;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	76 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	77
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	78 static PyObject *
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	79 internal_error_callback(const char *errors)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	80 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	81 if (errors == NULL \|\| strcmp(errors, "strict") == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	82 return ERROR_STRICT;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	83 else if (strcmp(errors, "ignore") == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	84 return ERROR_IGNORE;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	85 else if (strcmp(errors, "replace") == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	86 return ERROR_REPLACE;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	87 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	88 return PyString_FromString(errors);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	89 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	90
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	91 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	92 call_error_callback(PyObject errors, PyObject exc)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	93 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	94 PyObject args, cb, *r;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	95
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	96 assert(PyString_Check(errors));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	97 cb = PyCodec_LookupError(PyString_AS_STRING(errors));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	98 if (cb == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	99 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	100
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	101 args = PyTuple_New(1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	102 if (args == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	103 Py_DECREF(cb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	104 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	105 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	106
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	107 PyTuple_SET_ITEM(args, 0, exc);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	108 Py_INCREF(exc);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	109
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	110 r = PyObject_CallObject(cb, args);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	111 Py_DECREF(args);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	112 Py_DECREF(cb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	113 return r;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	114 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	115
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	116 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	117 codecctx_errors_get(MultibyteStatefulCodecContext *self)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	118 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	119 const char *errors;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	120
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	121 if (self->errors == ERROR_STRICT)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	122 errors = "strict";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	123 else if (self->errors == ERROR_IGNORE)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	124 errors = "ignore";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	125 else if (self->errors == ERROR_REPLACE)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	126 errors = "replace";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	127 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	128 Py_INCREF(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	129 return self->errors;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	130 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	131
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	132 return PyString_FromString(errors);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	133 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	134
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	135 static int
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	136 codecctx_errors_set(MultibyteStatefulCodecContext self, PyObject value,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	137 void *closure)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	138 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	139 PyObject *cb;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	140
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	141 if (!PyString_Check(value)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	142 PyErr_SetString(PyExc_TypeError, "errors must be a string");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	143 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	144 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	145
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	146 cb = internal_error_callback(PyString_AS_STRING(value));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	147 if (cb == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	148 return -1;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	149
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	150 ERROR_DECREF(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	151 self->errors = cb;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	152 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	153 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	154
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	155 /* This getset handlers list is used by all the stateful codec objects */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	156 static PyGetSetDef codecctx_getsets[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	157 {"errors", (getter)codecctx_errors_get,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	158 (setter)codecctx_errors_set,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	159 PyDoc_STR("how to treat errors")},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	160 {NULL,}
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	161 };
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	162
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	163 static int
36831 bac6bd343a12 - Modernize code to use Py_ssize_t more intensively. Hye-Shik Chang <hyeshik@gmail.com> parents: 36684 diff changeset	164 expand_encodebuffer(MultibyteEncodeBuffer *buf, Py_ssize_t esize)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	165 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	166 Py_ssize_t orgpos, orgsize, incsize;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	167
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	168 orgpos = (Py_ssize_t)((char *)buf->outbuf -
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	169 PyString_AS_STRING(buf->outobj));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	170 orgsize = PyString_GET_SIZE(buf->outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	171 incsize = (esize < (orgsize >> 1) ? (orgsize >> 1) \| 1 : esize);
45337 8cbe0bd93c28 Added checks for integer overflows, contributed by Google. Some are Martin v. Löwis <martin@v.loewis.de> parents: 41947 diff changeset	172
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	173 if (orgsize > PY_SSIZE_T_MAX - incsize)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	174 return -1;
45337 8cbe0bd93c28 Added checks for integer overflows, contributed by Google. Some are Martin v. Löwis <martin@v.loewis.de> parents: 41947 diff changeset	175
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	176 if (_PyString_Resize(&buf->outobj, orgsize + incsize) == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	177 return -1;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	178
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	179 buf->outbuf = (unsigned char *)PyString_AS_STRING(buf->outobj) +orgpos;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	180 buf->outbuf_end = (unsigned char *)PyString_AS_STRING(buf->outobj)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	181 + PyString_GET_SIZE(buf->outobj);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	182
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	183 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	184 }
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	185 #define REQUIRE_ENCODEBUFFER(buf, s) { \
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	186 if ((s) < 1 \|\| (buf)->outbuf + (s) > (buf)->outbuf_end) \
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	187 if (expand_encodebuffer(buf, s) == -1) \
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	188 goto errorexit; \
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	189 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	190
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	191 static int
36831 bac6bd343a12 - Modernize code to use Py_ssize_t more intensively. Hye-Shik Chang <hyeshik@gmail.com> parents: 36684 diff changeset	192 expand_decodebuffer(MultibyteDecodeBuffer *buf, Py_ssize_t esize)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	193 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	194 Py_ssize_t orgpos, orgsize;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	195
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	196 orgpos = (Py_ssize_t)(buf->outbuf - PyUnicode_AS_UNICODE(buf->outobj));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	197 orgsize = PyUnicode_GET_SIZE(buf->outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	198 if (PyUnicode_Resize(&buf->outobj, orgsize + (
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	199 esize < (orgsize >> 1) ? (orgsize >> 1) \| 1 : esize)) == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	200 return -1;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	201
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	202 buf->outbuf = PyUnicode_AS_UNICODE(buf->outobj) + orgpos;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	203 buf->outbuf_end = PyUnicode_AS_UNICODE(buf->outobj)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	204 + PyUnicode_GET_SIZE(buf->outobj);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	205
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	206 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	207 }
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	208 #define REQUIRE_DECODEBUFFER(buf, s) { \
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	209 if ((s) < 1 \|\| (buf)->outbuf + (s) > (buf)->outbuf_end) \
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	210 if (expand_decodebuffer(buf, s) == -1) \
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	211 goto errorexit; \
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	212 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	213
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	214
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	215 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	216 * MultibyteCodec object
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	217 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	218
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	219 static int
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	220 multibytecodec_encerror(MultibyteCodec *codec,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	221 MultibyteCodec_State *state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	222 MultibyteEncodeBuffer *buf,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	223 PyObject *errors, Py_ssize_t e)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	224 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	225 PyObject retobj = NULL, retstr = NULL, *tobj;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	226 Py_ssize_t retstrsize, newpos;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	227 Py_ssize_t esize, start, end;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	228 const char *reason;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	229
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	230 if (e > 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	231 reason = "illegal multibyte sequence";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	232 esize = e;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	233 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	234 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	235 switch (e) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	236 case MBERR_TOOSMALL:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	237 REQUIRE_ENCODEBUFFER(buf, -1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	238 return 0; /* retry it */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	239 case MBERR_TOOFEW:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	240 reason = "incomplete multibyte sequence";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	241 esize = (Py_ssize_t)(buf->inbuf_end - buf->inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	242 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	243 case MBERR_INTERNAL:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	244 PyErr_SetString(PyExc_RuntimeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	245 "internal codec error");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	246 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	247 default:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	248 PyErr_SetString(PyExc_RuntimeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	249 "unknown runtime error");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	250 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	251 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	252 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	253
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	254 if (errors == ERROR_REPLACE) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	255 const Py_UNICODE replchar = '?', *inbuf = &replchar;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	256 Py_ssize_t r;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	257
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	258 for (;;) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	259 Py_ssize_t outleft;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	260
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	261 outleft = (Py_ssize_t)(buf->outbuf_end - buf->outbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	262 r = codec->encode(state, codec->config, &inbuf, 1,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	263 &buf->outbuf, outleft, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	264 if (r == MBERR_TOOSMALL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	265 REQUIRE_ENCODEBUFFER(buf, -1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	266 continue;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	267 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	268 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	269 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	270 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	271
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	272 if (r != 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	273 REQUIRE_ENCODEBUFFER(buf, 1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	274 *buf->outbuf++ = '?';
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	275 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	276 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	277 if (errors == ERROR_IGNORE \|\| errors == ERROR_REPLACE) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	278 buf->inbuf += esize;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	279 return 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	280 }
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	281
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	282 start = (Py_ssize_t)(buf->inbuf - buf->inbuf_top);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	283 end = start + esize;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	284
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	285 /* use cached exception object if available */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	286 if (buf->excobj == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	287 buf->excobj = PyUnicodeEncodeError_Create(codec->encoding,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	288 buf->inbuf_top,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	289 buf->inbuf_end - buf->inbuf_top,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	290 start, end, reason);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	291 if (buf->excobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	292 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	293 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	294 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	295 if (PyUnicodeEncodeError_SetStart(buf->excobj, start) != 0 \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	296 PyUnicodeEncodeError_SetEnd(buf->excobj, end) != 0 \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	297 PyUnicodeEncodeError_SetReason(buf->excobj, reason) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	298 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	299
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	300 if (errors == ERROR_STRICT) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	301 PyCodec_StrictErrors(buf->excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	302 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	303 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	304
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	305 retobj = call_error_callback(errors, buf->excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	306 if (retobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	307 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	308
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	309 if (!PyTuple_Check(retobj) \|\| PyTuple_GET_SIZE(retobj) != 2 \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	310 !PyUnicode_Check((tobj = PyTuple_GET_ITEM(retobj, 0))) \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	311 !(PyInt_Check(PyTuple_GET_ITEM(retobj, 1)) \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	312 PyLong_Check(PyTuple_GET_ITEM(retobj, 1)))) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	313 PyErr_SetString(PyExc_TypeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	314 "encoding error handler must return "
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	315 "(unicode, int) tuple");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	316 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	317 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	318
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	319 {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	320 const Py_UNICODE *uraw = PyUnicode_AS_UNICODE(tobj);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	321
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	322 retstr = multibytecodec_encode(codec, state, &uraw,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	323 PyUnicode_GET_SIZE(tobj), ERROR_STRICT,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	324 MBENC_FLUSH);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	325 if (retstr == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	326 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	327 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	328
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	329 retstrsize = PyString_GET_SIZE(retstr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	330 REQUIRE_ENCODEBUFFER(buf, retstrsize);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	331
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	332 memcpy(buf->outbuf, PyString_AS_STRING(retstr), retstrsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	333 buf->outbuf += retstrsize;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	334
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	335 newpos = PyInt_AsSsize_t(PyTuple_GET_ITEM(retobj, 1));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	336 if (newpos < 0 && !PyErr_Occurred())
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	337 newpos += (Py_ssize_t)(buf->inbuf_end - buf->inbuf_top);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	338 if (newpos < 0 \|\| buf->inbuf_top + newpos > buf->inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	339 PyErr_Clear();
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	340 PyErr_Format(PyExc_IndexError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	341 "position %zd from error handler out of bounds",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	342 newpos);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	343 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	344 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	345 buf->inbuf = buf->inbuf_top + newpos;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	346
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	347 Py_DECREF(retobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	348 Py_DECREF(retstr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	349 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	350
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	351 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	352 Py_XDECREF(retobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	353 Py_XDECREF(retstr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	354 return -1;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	355 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	356
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	357 static int
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	358 multibytecodec_decerror(MultibyteCodec *codec,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	359 MultibyteCodec_State *state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	360 MultibyteDecodeBuffer *buf,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	361 PyObject *errors, Py_ssize_t e)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	362 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	363 PyObject retobj = NULL, retuni = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	364 Py_ssize_t retunisize, newpos;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	365 const char *reason;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	366 Py_ssize_t esize, start, end;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	367
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	368 if (e > 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	369 reason = "illegal multibyte sequence";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	370 esize = e;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	371 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	372 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	373 switch (e) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	374 case MBERR_TOOSMALL:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	375 REQUIRE_DECODEBUFFER(buf, -1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	376 return 0; /* retry it */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	377 case MBERR_TOOFEW:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	378 reason = "incomplete multibyte sequence";
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	379 esize = (Py_ssize_t)(buf->inbuf_end - buf->inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	380 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	381 case MBERR_INTERNAL:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	382 PyErr_SetString(PyExc_RuntimeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	383 "internal codec error");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	384 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	385 default:
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	386 PyErr_SetString(PyExc_RuntimeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	387 "unknown runtime error");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	388 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	389 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	390 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	391
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	392 if (errors == ERROR_REPLACE) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	393 REQUIRE_DECODEBUFFER(buf, 1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	394 *buf->outbuf++ = Py_UNICODE_REPLACEMENT_CHARACTER;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	395 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	396 if (errors == ERROR_IGNORE \|\| errors == ERROR_REPLACE) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	397 buf->inbuf += esize;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	398 return 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	399 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	400
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	401 start = (Py_ssize_t)(buf->inbuf - buf->inbuf_top);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	402 end = start + esize;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	403
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	404 /* use cached exception object if available */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	405 if (buf->excobj == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	406 buf->excobj = PyUnicodeDecodeError_Create(codec->encoding,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	407 (const char *)buf->inbuf_top,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	408 (Py_ssize_t)(buf->inbuf_end - buf->inbuf_top),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	409 start, end, reason);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	410 if (buf->excobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	411 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	412 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	413 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	414 if (PyUnicodeDecodeError_SetStart(buf->excobj, start) \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	415 PyUnicodeDecodeError_SetEnd(buf->excobj, end) \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	416 PyUnicodeDecodeError_SetReason(buf->excobj, reason))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	417 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	418
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	419 if (errors == ERROR_STRICT) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	420 PyCodec_StrictErrors(buf->excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	421 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	422 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	423
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	424 retobj = call_error_callback(errors, buf->excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	425 if (retobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	426 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	427
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	428 if (!PyTuple_Check(retobj) \|\| PyTuple_GET_SIZE(retobj) != 2 \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	429 !PyUnicode_Check((retuni = PyTuple_GET_ITEM(retobj, 0))) \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	430 !(PyInt_Check(PyTuple_GET_ITEM(retobj, 1)) \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	431 PyLong_Check(PyTuple_GET_ITEM(retobj, 1)))) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	432 PyErr_SetString(PyExc_TypeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	433 "decoding error handler must return "
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	434 "(unicode, int) tuple");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	435 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	436 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	437
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	438 retunisize = PyUnicode_GET_SIZE(retuni);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	439 if (retunisize > 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	440 REQUIRE_DECODEBUFFER(buf, retunisize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	441 memcpy((char *)buf->outbuf, PyUnicode_AS_DATA(retuni),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	442 retunisize * Py_UNICODE_SIZE);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	443 buf->outbuf += retunisize;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	444 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	445
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	446 newpos = PyInt_AsSsize_t(PyTuple_GET_ITEM(retobj, 1));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	447 if (newpos < 0 && !PyErr_Occurred())
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	448 newpos += (Py_ssize_t)(buf->inbuf_end - buf->inbuf_top);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	449 if (newpos < 0 \|\| buf->inbuf_top + newpos > buf->inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	450 PyErr_Clear();
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	451 PyErr_Format(PyExc_IndexError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	452 "position %zd from error handler out of bounds",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	453 newpos);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	454 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	455 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	456 buf->inbuf = buf->inbuf_top + newpos;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	457 Py_DECREF(retobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	458 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	459
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	460 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	461 Py_XDECREF(retobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	462 return -1;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	463 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	464
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	465 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	466 multibytecodec_encode(MultibyteCodec *codec,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	467 MultibyteCodec_State *state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	468 const Py_UNICODE **data, Py_ssize_t datalen,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	469 PyObject *errors, int flags)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	470 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	471 MultibyteEncodeBuffer buf;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	472 Py_ssize_t finalsize, r = 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	473
70350 e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	474 if (datalen == 0 && !(flags & MBENC_RESET))
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	475 return PyString_FromString("");
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	476
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	477 buf.excobj = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	478 buf.inbuf = buf.inbuf_top = *data;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	479 buf.inbuf_end = buf.inbuf_top + datalen;
45337 8cbe0bd93c28 Added checks for integer overflows, contributed by Google. Some are Martin v. Löwis <martin@v.loewis.de> parents: 41947 diff changeset	480
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	481 if (datalen > (PY_SSIZE_T_MAX - 16) / 2) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	482 PyErr_NoMemory();
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	483 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	484 }
45337 8cbe0bd93c28 Added checks for integer overflows, contributed by Google. Some are Martin v. Löwis <martin@v.loewis.de> parents: 41947 diff changeset	485
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	486 buf.outobj = PyString_FromStringAndSize(NULL, datalen * 2 + 16);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	487 if (buf.outobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	488 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	489 buf.outbuf = (unsigned char *)PyString_AS_STRING(buf.outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	490 buf.outbuf_end = buf.outbuf + PyString_GET_SIZE(buf.outobj);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	491
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	492 while (buf.inbuf < buf.inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	493 Py_ssize_t inleft, outleft;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	494
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	495 /* we don't reuse inleft and outleft here.
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	496 * error callbacks can relocate the cursor anywhere on buffer*/
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	497 inleft = (Py_ssize_t)(buf.inbuf_end - buf.inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	498 outleft = (Py_ssize_t)(buf.outbuf_end - buf.outbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	499 r = codec->encode(state, codec->config, &buf.inbuf, inleft,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	500 &buf.outbuf, outleft, flags);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	501 if ((r == 0) \|\| (r == MBERR_TOOFEW && !(flags & MBENC_FLUSH)))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	502 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	503 else if (multibytecodec_encerror(codec, state, &buf, errors,r))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	504 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	505 else if (r == MBERR_TOOFEW)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	506 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	507 }
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	508
70350 e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	509 if (codec->encreset != NULL && (flags & MBENC_RESET))
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	510 for (;;) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	511 Py_ssize_t outleft;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	512
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	513 outleft = (Py_ssize_t)(buf.outbuf_end - buf.outbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	514 r = codec->encreset(state, codec->config, &buf.outbuf,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	515 outleft);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	516 if (r == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	517 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	518 else if (multibytecodec_encerror(codec, state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	519 &buf, errors, r))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	520 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	521 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	522
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	523 finalsize = (Py_ssize_t)((char *)buf.outbuf -
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	524 PyString_AS_STRING(buf.outobj));
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	525
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	526 if (finalsize != PyString_GET_SIZE(buf.outobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	527 if (_PyString_Resize(&buf.outobj, finalsize) == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	528 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	529
61845 14d23bddf793 Merged revisions 81454 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61465 diff changeset	530 *data = buf.inbuf;
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	531 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	532 return buf.outobj;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	533
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	534 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	535 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	536 Py_XDECREF(buf.outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	537 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	538 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	539
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	540 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	541 MultibyteCodec_Encode(MultibyteCodecObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	542 PyObject args, PyObject kwargs)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	543 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	544 MultibyteCodec_State state;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	545 Py_UNICODE *data;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	546 PyObject errorcb, r, arg, ucvt;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	547 const char *errors = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	548 Py_ssize_t datalen;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	549
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	550 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|z:encode",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	551 codeckwarglist, &arg, &errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	552 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	553
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	554 if (PyUnicode_Check(arg))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	555 ucvt = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	556 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	557 arg = ucvt = PyObject_Unicode(arg);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	558 if (arg == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	559 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	560 else if (!PyUnicode_Check(arg)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	561 PyErr_SetString(PyExc_TypeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	562 "couldn't convert the object to unicode.");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	563 Py_DECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	564 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	565 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	566 }
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	567
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	568 data = PyUnicode_AS_UNICODE(arg);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	569 datalen = PyUnicode_GET_SIZE(arg);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	570
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	571 errorcb = internal_error_callback(errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	572 if (errorcb == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	573 Py_XDECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	574 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	575 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	576
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	577 if (self->codec->encinit != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	578 self->codec->encinit(&state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	579 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	580 r = multibytecodec_encode(self->codec, &state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	581 (const Py_UNICODE **)&data, datalen, errorcb,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	582 MBENC_FLUSH \| MBENC_RESET);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	583 if (r == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	584 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	585
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	586 ERROR_DECREF(errorcb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	587 Py_XDECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	588 return make_tuple(r, datalen);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	589
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	590 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	591 ERROR_DECREF(errorcb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	592 Py_XDECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	593 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	594 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	595
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	596 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	597 MultibyteCodec_Decode(MultibyteCodecObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	598 PyObject args, PyObject kwargs)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	599 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	600 MultibyteCodec_State state;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	601 MultibyteDecodeBuffer buf;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	602 PyObject *errorcb;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	603 Py_buffer pdata;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	604 const char data, errors = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	605 Py_ssize_t datalen, finalsize;
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	606
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	607 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s*\|z:decode",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	608 codeckwarglist, &pdata, &errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	609 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	610 data = pdata.buf;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	611 datalen = pdata.len;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	612
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	613 errorcb = internal_error_callback(errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	614 if (errorcb == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	615 PyBuffer_Release(&pdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	616 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	617 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	618
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	619 if (datalen == 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	620 PyBuffer_Release(&pdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	621 ERROR_DECREF(errorcb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	622 return make_tuple(PyUnicode_FromUnicode(NULL, 0), 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	623 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	624
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	625 buf.excobj = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	626 buf.inbuf = buf.inbuf_top = (unsigned char *)data;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	627 buf.inbuf_end = buf.inbuf_top + datalen;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	628 buf.outobj = PyUnicode_FromUnicode(NULL, datalen);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	629 if (buf.outobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	630 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	631 buf.outbuf = PyUnicode_AS_UNICODE(buf.outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	632 buf.outbuf_end = buf.outbuf + PyUnicode_GET_SIZE(buf.outobj);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	633
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	634 if (self->codec->decinit != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	635 self->codec->decinit(&state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	636 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	637
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	638 while (buf.inbuf < buf.inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	639 Py_ssize_t inleft, outleft, r;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	640
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	641 inleft = (Py_ssize_t)(buf.inbuf_end - buf.inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	642 outleft = (Py_ssize_t)(buf.outbuf_end - buf.outbuf);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	643
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	644 r = self->codec->decode(&state, self->codec->config,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	645 &buf.inbuf, inleft, &buf.outbuf, outleft);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	646 if (r == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	647 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	648 else if (multibytecodec_decerror(self->codec, &state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	649 &buf, errorcb, r))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	650 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	651 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	652
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	653 finalsize = (Py_ssize_t)(buf.outbuf -
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	654 PyUnicode_AS_UNICODE(buf.outobj));
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	655
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	656 if (finalsize != PyUnicode_GET_SIZE(buf.outobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	657 if (PyUnicode_Resize(&buf.outobj, finalsize) == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	658 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	659
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	660 PyBuffer_Release(&pdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	661 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	662 ERROR_DECREF(errorcb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	663 return make_tuple(buf.outobj, datalen);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	664
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	665 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	666 PyBuffer_Release(&pdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	667 ERROR_DECREF(errorcb);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	668 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	669 Py_XDECREF(buf.outobj);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	670
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	671 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	672 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	673
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	674 static struct PyMethodDef multibytecodec_methods[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	675 {"encode", (PyCFunction)MultibyteCodec_Encode,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	676 METH_VARARGS \| METH_KEYWORDS,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	677 MultibyteCodec_Encode__doc__},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	678 {"decode", (PyCFunction)MultibyteCodec_Decode,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	679 METH_VARARGS \| METH_KEYWORDS,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	680 MultibyteCodec_Decode__doc__},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	681 {NULL, NULL},
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	682 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	683
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	684 static void
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	685 multibytecodec_dealloc(MultibyteCodecObject *self)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	686 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	687 PyObject_Del(self);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	688 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	689
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	690 static PyTypeObject MultibyteCodec_Type = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	691 PyVarObject_HEAD_INIT(NULL, 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	692 "MultibyteCodec", /* tp_name */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	693 sizeof(MultibyteCodecObject), /* tp_basicsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	694 0, /* tp_itemsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	695 /* methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	696 (destructor)multibytecodec_dealloc, /* tp_dealloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	697 0, /* tp_print */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	698 0, /* tp_getattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	699 0, /* tp_setattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	700 0, /* tp_compare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	701 0, /* tp_repr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	702 0, /* tp_as_number */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	703 0, /* tp_as_sequence */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	704 0, /* tp_as_mapping */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	705 0, /* tp_hash */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	706 0, /* tp_call */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	707 0, /* tp_str */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	708 PyObject_GenericGetAttr, /* tp_getattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	709 0, /* tp_setattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	710 0, /* tp_as_buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	711 Py_TPFLAGS_DEFAULT, /* tp_flags */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	712 0, /* tp_doc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	713 0, /* tp_traverse */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	714 0, /* tp_clear */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	715 0, /* tp_richcompare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	716 0, /* tp_weaklistoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	717 0, /* tp_iter */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	718 0, /* tp_iterext */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	719 multibytecodec_methods, /* tp_methods */
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	720 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	721
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	722
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	723 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	724 * Utility functions for stateful codec mechanism
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	725 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	726
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	727 #define STATEFUL_DCTX(o) ((MultibyteStatefulDecoderContext *)(o))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	728 #define STATEFUL_ECTX(o) ((MultibyteStatefulEncoderContext *)(o))
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	729
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	730 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	731 encoder_encode_stateful(MultibyteStatefulEncoderContext *ctx,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	732 PyObject *unistr, int final)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	733 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	734 PyObject ucvt, r = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	735 Py_UNICODE inbuf, inbuf_end, *inbuf_tmp = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	736 Py_ssize_t datalen, origpending;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	737
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	738 if (PyUnicode_Check(unistr))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	739 ucvt = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	740 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	741 unistr = ucvt = PyObject_Unicode(unistr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	742 if (unistr == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	743 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	744 else if (!PyUnicode_Check(unistr)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	745 PyErr_SetString(PyExc_TypeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	746 "couldn't convert the object to unicode.");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	747 Py_DECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	748 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	749 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	750 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	751
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	752 datalen = PyUnicode_GET_SIZE(unistr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	753 origpending = ctx->pendingsize;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	754
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	755 if (origpending > 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	756 if (datalen > PY_SSIZE_T_MAX - ctx->pendingsize) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	757 PyErr_NoMemory();
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	758 /* inbuf_tmp == NULL */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	759 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	760 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	761 inbuf_tmp = PyMem_New(Py_UNICODE, datalen + ctx->pendingsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	762 if (inbuf_tmp == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	763 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	764 memcpy(inbuf_tmp, ctx->pending,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	765 Py_UNICODE_SIZE * ctx->pendingsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	766 memcpy(inbuf_tmp + ctx->pendingsize,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	767 PyUnicode_AS_UNICODE(unistr),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	768 Py_UNICODE_SIZE * datalen);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	769 datalen += ctx->pendingsize;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	770 ctx->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	771 inbuf = inbuf_tmp;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	772 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	773 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	774 inbuf = (Py_UNICODE *)PyUnicode_AS_UNICODE(unistr);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	775
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	776 inbuf_end = inbuf + datalen;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	777
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	778 r = multibytecodec_encode(ctx->codec, &ctx->state,
70350 e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	779 (const Py_UNICODE **)&inbuf, datalen,
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	780 ctx->errors, final ? MBENC_FLUSH \| MBENC_RESET : 0);
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	781 if (r == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	782 /* recover the original pending buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	783 if (origpending > 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	784 memcpy(ctx->pending, inbuf_tmp,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	785 Py_UNICODE_SIZE * origpending);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	786 ctx->pendingsize = origpending;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	787 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	788 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	789
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	790 if (inbuf < inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	791 ctx->pendingsize = (Py_ssize_t)(inbuf_end - inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	792 if (ctx->pendingsize > MAXENCPENDING) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	793 /* normal codecs can't reach here */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	794 ctx->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	795 PyErr_SetString(PyExc_UnicodeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	796 "pending buffer overflow");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	797 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	798 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	799 memcpy(ctx->pending, inbuf,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	800 ctx->pendingsize * Py_UNICODE_SIZE);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	801 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	802
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	803 if (inbuf_tmp != NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	804 PyMem_Del(inbuf_tmp);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	805 Py_XDECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	806 return r;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	807
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	808 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	809 if (inbuf_tmp != NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	810 PyMem_Del(inbuf_tmp);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	811 Py_XDECREF(r);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	812 Py_XDECREF(ucvt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	813 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	814 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	815
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	816 static int
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	817 decoder_append_pending(MultibyteStatefulDecoderContext *ctx,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	818 MultibyteDecodeBuffer *buf)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	819 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	820 Py_ssize_t npendings;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	821
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	822 npendings = (Py_ssize_t)(buf->inbuf_end - buf->inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	823 if (npendings + ctx->pendingsize > MAXDECPENDING \|\|
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	824 npendings > PY_SSIZE_T_MAX - ctx->pendingsize) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	825 PyErr_SetString(PyExc_UnicodeError, "pending buffer overflow");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	826 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	827 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	828 memcpy(ctx->pending + ctx->pendingsize, buf->inbuf, npendings);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	829 ctx->pendingsize += npendings;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	830 return 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	831 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	832
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	833 static int
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	834 decoder_prepare_buffer(MultibyteDecodeBuffer buf, const char data,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	835 Py_ssize_t size)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	836 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	837 buf->inbuf = buf->inbuf_top = (const unsigned char *)data;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	838 buf->inbuf_end = buf->inbuf_top + size;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	839 if (buf->outobj == NULL) { /* only if outobj is not allocated yet */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	840 buf->outobj = PyUnicode_FromUnicode(NULL, size);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	841 if (buf->outobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	842 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	843 buf->outbuf = PyUnicode_AS_UNICODE(buf->outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	844 buf->outbuf_end = buf->outbuf +
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	845 PyUnicode_GET_SIZE(buf->outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	846 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	847
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	848 return 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	849 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	850
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	851 static int
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	852 decoder_feed_buffer(MultibyteStatefulDecoderContext *ctx,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	853 MultibyteDecodeBuffer *buf)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	854 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	855 while (buf->inbuf < buf->inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	856 Py_ssize_t inleft, outleft;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	857 Py_ssize_t r;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	858
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	859 inleft = (Py_ssize_t)(buf->inbuf_end - buf->inbuf);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	860 outleft = (Py_ssize_t)(buf->outbuf_end - buf->outbuf);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	861
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	862 r = ctx->codec->decode(&ctx->state, ctx->codec->config,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	863 &buf->inbuf, inleft, &buf->outbuf, outleft);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	864 if (r == 0 \|\| r == MBERR_TOOFEW)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	865 break;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	866 else if (multibytecodec_decerror(ctx->codec, &ctx->state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	867 buf, ctx->errors, r))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	868 return -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	869 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	870 return 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	871 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	872
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	873
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	874 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	875 * MultibyteIncrementalEncoder object
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	876 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	877
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	878 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	879 mbiencoder_encode(MultibyteIncrementalEncoderObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	880 PyObject args, PyObject kwargs)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	881 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	882 PyObject *data;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	883 int final = 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	884
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	885 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|i:encode",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	886 incrementalkwarglist, &data, &final))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	887 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	888
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	889 return encoder_encode_stateful(STATEFUL_ECTX(self), data, final);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	890 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	891
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	892 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	893 mbiencoder_reset(MultibyteIncrementalEncoderObject *self)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	894 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	895 if (self->codec->decreset != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	896 self->codec->decreset(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	897 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	898 self->pendingsize = 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	899
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	900 Py_RETURN_NONE;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	901 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	902
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	903 static struct PyMethodDef mbiencoder_methods[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	904 {"encode", (PyCFunction)mbiencoder_encode,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	905 METH_VARARGS \| METH_KEYWORDS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	906 {"reset", (PyCFunction)mbiencoder_reset,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	907 METH_NOARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	908 {NULL, NULL},
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	909 };
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	910
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	911 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	912 mbiencoder_new(PyTypeObject type, PyObject args, PyObject *kwds)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	913 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	914 MultibyteIncrementalEncoderObject *self;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	915 PyObject *codec = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	916 char *errors = NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	917
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	918 if (!PyArg_ParseTupleAndKeywords(args, kwds, "\|s:IncrementalEncoder",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	919 incnewkwarglist, &errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	920 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	921
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	922 self = (MultibyteIncrementalEncoderObject *)type->tp_alloc(type, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	923 if (self == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	924 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	925
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	926 codec = PyObject_GetAttrString((PyObject *)type, "codec");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	927 if (codec == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	928 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	929 if (!MultibyteCodec_Check(codec)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	930 PyErr_SetString(PyExc_TypeError, "codec is unexpected type");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	931 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	932 }
37233 0098e7264eaa Fix reference leaks introduced by the recent incremental codec Hye-Shik Chang <hyeshik@gmail.com> parents: 37224 diff changeset	933
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	934 self->codec = ((MultibyteCodecObject *)codec)->codec;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	935 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	936 self->errors = internal_error_callback(errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	937 if (self->errors == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	938 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	939 if (self->codec->encinit != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	940 self->codec->encinit(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	941 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	942
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	943 Py_DECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	944 return (PyObject *)self;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	945
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	946 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	947 Py_XDECREF(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	948 Py_XDECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	949 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	950 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	951
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	952 static int
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	953 mbiencoder_init(PyObject self, PyObject args, PyObject *kwds)
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	954 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	955 return 0;
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	956 }
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	957
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	958 static int
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	959 mbiencoder_traverse(MultibyteIncrementalEncoderObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	960 visitproc visit, void *arg)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	961 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	962 if (ERROR_ISCUSTOM(self->errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	963 Py_VISIT(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	964 return 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	965 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	966
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	967 static void
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	968 mbiencoder_dealloc(MultibyteIncrementalEncoderObject *self)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	969 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	970 PyObject_GC_UnTrack(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	971 ERROR_DECREF(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	972 Py_TYPE(self)->tp_free(self);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	973 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	974
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	975 static PyTypeObject MultibyteIncrementalEncoder_Type = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	976 PyVarObject_HEAD_INIT(NULL, 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	977 "MultibyteIncrementalEncoder", /* tp_name */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	978 sizeof(MultibyteIncrementalEncoderObject), /* tp_basicsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	979 0, /* tp_itemsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	980 /* methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	981 (destructor)mbiencoder_dealloc, /* tp_dealloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	982 0, /* tp_print */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	983 0, /* tp_getattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	984 0, /* tp_setattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	985 0, /* tp_compare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	986 0, /* tp_repr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	987 0, /* tp_as_number */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	988 0, /* tp_as_sequence */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	989 0, /* tp_as_mapping */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	990 0, /* tp_hash */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	991 0, /* tp_call */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	992 0, /* tp_str */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	993 PyObject_GenericGetAttr, /* tp_getattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	994 0, /* tp_setattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	995 0, /* tp_as_buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	996 Py_TPFLAGS_DEFAULT \| Py_TPFLAGS_HAVE_GC
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	997 \| Py_TPFLAGS_BASETYPE, /* tp_flags */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	998 0, /* tp_doc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	999 (traverseproc)mbiencoder_traverse, /* tp_traverse */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1000 0, /* tp_clear */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1001 0, /* tp_richcompare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1002 0, /* tp_weaklistoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1003 0, /* tp_iter */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1004 0, /* tp_iterext */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1005 mbiencoder_methods, /* tp_methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1006 0, /* tp_members */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1007 codecctx_getsets, /* tp_getset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1008 0, /* tp_base */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1009 0, /* tp_dict */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1010 0, /* tp_descr_get */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1011 0, /* tp_descr_set */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1012 0, /* tp_dictoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1013 mbiencoder_init, /* tp_init */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1014 0, /* tp_alloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1015 mbiencoder_new, /* tp_new */
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1016 };
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1017
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1018
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1019 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1020 * MultibyteIncrementalDecoder object
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1021 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1022
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1023 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1024 mbidecoder_decode(MultibyteIncrementalDecoderObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1025 PyObject args, PyObject kwargs)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1026 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1027 MultibyteDecodeBuffer buf;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1028 char data, wdata = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1029 Py_buffer pdata;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1030 Py_ssize_t wsize, finalsize = 0, size, origpending;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1031 int final = 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1032
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1033 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s*\|i:decode",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1034 incrementalkwarglist, &pdata, &final))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1035 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1036 data = pdata.buf;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1037 size = pdata.len;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1038
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1039 buf.outobj = buf.excobj = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1040 origpending = self->pendingsize;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1041
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1042 if (self->pendingsize == 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1043 wsize = size;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1044 wdata = data;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1045 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1046 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1047 if (size > PY_SSIZE_T_MAX - self->pendingsize) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1048 PyErr_NoMemory();
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1049 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1050 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1051 wsize = size + self->pendingsize;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1052 wdata = PyMem_Malloc(wsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1053 if (wdata == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1054 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1055 memcpy(wdata, self->pending, self->pendingsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1056 memcpy(wdata + self->pendingsize, data, size);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1057 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1058 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1059
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1060 if (decoder_prepare_buffer(&buf, wdata, wsize) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1061 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1062
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1063 if (decoder_feed_buffer(STATEFUL_DCTX(self), &buf))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1064 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1065
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1066 if (final && buf.inbuf < buf.inbuf_end) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1067 if (multibytecodec_decerror(self->codec, &self->state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1068 &buf, self->errors, MBERR_TOOFEW)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1069 /* recover the original pending buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1070 memcpy(self->pending, wdata, origpending);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1071 self->pendingsize = origpending;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1072 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1073 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1074 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1075
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1076 if (buf.inbuf < buf.inbuf_end) { /* pending sequence still exists */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1077 if (decoder_append_pending(STATEFUL_DCTX(self), &buf) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1078 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1079 }
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1080
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1081 finalsize = (Py_ssize_t)(buf.outbuf - PyUnicode_AS_UNICODE(buf.outobj));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1082 if (finalsize != PyUnicode_GET_SIZE(buf.outobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1083 if (PyUnicode_Resize(&buf.outobj, finalsize) == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1084 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1085
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1086 PyBuffer_Release(&pdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1087 if (wdata != data)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1088 PyMem_Del(wdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1089 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1090 return buf.outobj;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1091
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1092 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1093 PyBuffer_Release(&pdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1094 if (wdata != NULL && wdata != data)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1095 PyMem_Del(wdata);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1096 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1097 Py_XDECREF(buf.outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1098 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1099 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1100
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1101 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1102 mbidecoder_reset(MultibyteIncrementalDecoderObject *self)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1103 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1104 if (self->codec->decreset != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1105 self->codec->decreset(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1106 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1107 self->pendingsize = 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1108
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1109 Py_RETURN_NONE;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1110 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1111
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1112 static struct PyMethodDef mbidecoder_methods[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1113 {"decode", (PyCFunction)mbidecoder_decode,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1114 METH_VARARGS \| METH_KEYWORDS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1115 {"reset", (PyCFunction)mbidecoder_reset,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1116 METH_NOARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1117 {NULL, NULL},
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1118 };
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1119
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1120 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1121 mbidecoder_new(PyTypeObject type, PyObject args, PyObject *kwds)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1122 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1123 MultibyteIncrementalDecoderObject *self;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1124 PyObject *codec = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1125 char *errors = NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1126
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1127 if (!PyArg_ParseTupleAndKeywords(args, kwds, "\|s:IncrementalDecoder",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1128 incnewkwarglist, &errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1129 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1130
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1131 self = (MultibyteIncrementalDecoderObject *)type->tp_alloc(type, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1132 if (self == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1133 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1134
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1135 codec = PyObject_GetAttrString((PyObject *)type, "codec");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1136 if (codec == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1137 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1138 if (!MultibyteCodec_Check(codec)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1139 PyErr_SetString(PyExc_TypeError, "codec is unexpected type");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1140 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1141 }
37233 0098e7264eaa Fix reference leaks introduced by the recent incremental codec Hye-Shik Chang <hyeshik@gmail.com> parents: 37224 diff changeset	1142
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1143 self->codec = ((MultibyteCodecObject *)codec)->codec;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1144 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1145 self->errors = internal_error_callback(errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1146 if (self->errors == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1147 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1148 if (self->codec->decinit != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1149 self->codec->decinit(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1150 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1151
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1152 Py_DECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1153 return (PyObject *)self;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1154
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1155 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1156 Py_XDECREF(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1157 Py_XDECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1158 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1159 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1160
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1161 static int
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1162 mbidecoder_init(PyObject self, PyObject args, PyObject *kwds)
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1163 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1164 return 0;
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1165 }
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1166
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1167 static int
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1168 mbidecoder_traverse(MultibyteIncrementalDecoderObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1169 visitproc visit, void *arg)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1170 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1171 if (ERROR_ISCUSTOM(self->errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1172 Py_VISIT(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1173 return 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1174 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1175
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1176 static void
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1177 mbidecoder_dealloc(MultibyteIncrementalDecoderObject *self)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1178 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1179 PyObject_GC_UnTrack(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1180 ERROR_DECREF(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1181 Py_TYPE(self)->tp_free(self);
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1182 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1183
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1184 static PyTypeObject MultibyteIncrementalDecoder_Type = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1185 PyVarObject_HEAD_INIT(NULL, 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1186 "MultibyteIncrementalDecoder", /* tp_name */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1187 sizeof(MultibyteIncrementalDecoderObject), /* tp_basicsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1188 0, /* tp_itemsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1189 /* methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1190 (destructor)mbidecoder_dealloc, /* tp_dealloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1191 0, /* tp_print */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1192 0, /* tp_getattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1193 0, /* tp_setattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1194 0, /* tp_compare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1195 0, /* tp_repr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1196 0, /* tp_as_number */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1197 0, /* tp_as_sequence */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1198 0, /* tp_as_mapping */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1199 0, /* tp_hash */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1200 0, /* tp_call */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1201 0, /* tp_str */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1202 PyObject_GenericGetAttr, /* tp_getattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1203 0, /* tp_setattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1204 0, /* tp_as_buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1205 Py_TPFLAGS_DEFAULT \| Py_TPFLAGS_HAVE_GC
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1206 \| Py_TPFLAGS_BASETYPE, /* tp_flags */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1207 0, /* tp_doc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1208 (traverseproc)mbidecoder_traverse, /* tp_traverse */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1209 0, /* tp_clear */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1210 0, /* tp_richcompare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1211 0, /* tp_weaklistoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1212 0, /* tp_iter */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1213 0, /* tp_iterext */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1214 mbidecoder_methods, /* tp_methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1215 0, /* tp_members */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1216 codecctx_getsets, /* tp_getset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1217 0, /* tp_base */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1218 0, /* tp_dict */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1219 0, /* tp_descr_get */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1220 0, /* tp_descr_set */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1221 0, /* tp_dictoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1222 mbidecoder_init, /* tp_init */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1223 0, /* tp_alloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1224 mbidecoder_new, /* tp_new */
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1225 };
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1226
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1227
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1228 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1229 * MultibyteStreamReader object
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1230 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1231
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1232 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1233 mbstreamreader_iread(MultibyteStreamReaderObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1234 const char *method, Py_ssize_t sizehint)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1235 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1236 MultibyteDecodeBuffer buf;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1237 PyObject *cres;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1238 Py_ssize_t rsize, finalsize = 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1239
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1240 if (sizehint == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1241 return PyUnicode_FromUnicode(NULL, 0);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1242
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1243 buf.outobj = buf.excobj = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1244 cres = NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1245
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1246 for (;;) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1247 int endoffile;
41947 b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it Hye-Shik Chang <hyeshik@gmail.com> parents: 38489 diff changeset	1248
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1249 if (sizehint < 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1250 cres = PyObject_CallMethod(self->stream,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1251 (char *)method, NULL);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1252 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1253 cres = PyObject_CallMethod(self->stream,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1254 (char *)method, "i", sizehint);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1255 if (cres == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1256 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1257
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1258 if (!PyString_Check(cres)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1259 PyErr_SetString(PyExc_TypeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1260 "stream function returned a "
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1261 "non-string object");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1262 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1263 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1264
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1265 endoffile = (PyString_GET_SIZE(cres) == 0);
41947 b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it Hye-Shik Chang <hyeshik@gmail.com> parents: 38489 diff changeset	1266
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1267 if (self->pendingsize > 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1268 PyObject *ctr;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1269 char *ctrdata;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1270
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1271 if (PyString_GET_SIZE(cres) > PY_SSIZE_T_MAX - self->pendingsize) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1272 PyErr_NoMemory();
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1273 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1274 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1275 rsize = PyString_GET_SIZE(cres) + self->pendingsize;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1276 ctr = PyString_FromStringAndSize(NULL, rsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1277 if (ctr == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1278 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1279 ctrdata = PyString_AS_STRING(ctr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1280 memcpy(ctrdata, self->pending, self->pendingsize);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1281 memcpy(ctrdata + self->pendingsize,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1282 PyString_AS_STRING(cres),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1283 PyString_GET_SIZE(cres));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1284 Py_DECREF(cres);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1285 cres = ctr;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1286 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1287 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1288
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1289 rsize = PyString_GET_SIZE(cres);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1290 if (decoder_prepare_buffer(&buf, PyString_AS_STRING(cres),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1291 rsize) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1292 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1293
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1294 if (rsize > 0 && decoder_feed_buffer(
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1295 (MultibyteStatefulDecoderContext *)self, &buf))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1296 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1297
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1298 if (endoffile \|\| sizehint < 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1299 if (buf.inbuf < buf.inbuf_end &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1300 multibytecodec_decerror(self->codec, &self->state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1301 &buf, self->errors, MBERR_TOOFEW))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1302 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1303 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1304
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1305 if (buf.inbuf < buf.inbuf_end) { /* pending sequence exists */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1306 if (decoder_append_pending(STATEFUL_DCTX(self),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1307 &buf) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1308 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1309 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1310
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1311 finalsize = (Py_ssize_t)(buf.outbuf -
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1312 PyUnicode_AS_UNICODE(buf.outobj));
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1313 Py_DECREF(cres);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1314 cres = NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1315
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1316 if (sizehint < 0 \|\| finalsize != 0 \|\| rsize == 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1317 break;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1318
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1319 sizehint = 1; /* read 1 more byte and retry */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1320 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1321
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1322 if (finalsize != PyUnicode_GET_SIZE(buf.outobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1323 if (PyUnicode_Resize(&buf.outobj, finalsize) == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1324 goto errorexit;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1325
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1326 Py_XDECREF(cres);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1327 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1328 return buf.outobj;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1329
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1330 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1331 Py_XDECREF(cres);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1332 Py_XDECREF(buf.excobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1333 Py_XDECREF(buf.outobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1334 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1335 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1336
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1337 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1338 mbstreamreader_read(MultibyteStreamReaderObject self, PyObject args)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1339 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1340 PyObject *sizeobj = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1341 Py_ssize_t size;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1342
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1343 if (!PyArg_UnpackTuple(args, "read", 0, 1, &sizeobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1344 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1345
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1346 if (sizeobj == Py_None \|\| sizeobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1347 size = -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1348 else if (PyInt_Check(sizeobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1349 size = PyInt_AsSsize_t(sizeobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1350 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1351 PyErr_SetString(PyExc_TypeError, "arg 1 must be an integer");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1352 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1353 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1354
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1355 return mbstreamreader_iread(self, "read", size);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1356 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1357
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1358 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1359 mbstreamreader_readline(MultibyteStreamReaderObject self, PyObject args)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1360 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1361 PyObject *sizeobj = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1362 Py_ssize_t size;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1363
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1364 if (!PyArg_UnpackTuple(args, "readline", 0, 1, &sizeobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1365 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1366
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1367 if (sizeobj == Py_None \|\| sizeobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1368 size = -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1369 else if (PyInt_Check(sizeobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1370 size = PyInt_AsSsize_t(sizeobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1371 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1372 PyErr_SetString(PyExc_TypeError, "arg 1 must be an integer");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1373 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1374 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1375
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1376 return mbstreamreader_iread(self, "readline", size);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1377 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1378
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1379 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1380 mbstreamreader_readlines(MultibyteStreamReaderObject self, PyObject args)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1381 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1382 PyObject sizehintobj = NULL, r, *sr;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1383 Py_ssize_t sizehint;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1384
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1385 if (!PyArg_UnpackTuple(args, "readlines", 0, 1, &sizehintobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1386 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1387
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1388 if (sizehintobj == Py_None \|\| sizehintobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1389 sizehint = -1;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1390 else if (PyInt_Check(sizehintobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1391 sizehint = PyInt_AsSsize_t(sizehintobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1392 else {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1393 PyErr_SetString(PyExc_TypeError, "arg 1 must be an integer");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1394 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1395 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1396
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1397 r = mbstreamreader_iread(self, "read", sizehint);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1398 if (r == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1399 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1400
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1401 sr = PyUnicode_Splitlines(r, 1);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1402 Py_DECREF(r);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1403 return sr;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1404 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1405
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1406 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1407 mbstreamreader_reset(MultibyteStreamReaderObject *self)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1408 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1409 if (self->codec->decreset != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1410 self->codec->decreset(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1411 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1412 self->pendingsize = 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1413
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1414 Py_RETURN_NONE;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1415 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1416
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1417 static struct PyMethodDef mbstreamreader_methods[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1418 {"read", (PyCFunction)mbstreamreader_read,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1419 METH_VARARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1420 {"readline", (PyCFunction)mbstreamreader_readline,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1421 METH_VARARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1422 {"readlines", (PyCFunction)mbstreamreader_readlines,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1423 METH_VARARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1424 {"reset", (PyCFunction)mbstreamreader_reset,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1425 METH_NOARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1426 {NULL, NULL},
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1427 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1428
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1429 static PyMemberDef mbstreamreader_members[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1430 {"stream", T_OBJECT,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1431 offsetof(MultibyteStreamReaderObject, stream),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1432 READONLY, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1433 {NULL,}
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1434 };
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1435
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1436 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1437 mbstreamreader_new(PyTypeObject type, PyObject args, PyObject *kwds)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1438 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1439 MultibyteStreamReaderObject *self;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1440 PyObject stream, codec = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1441 char *errors = NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1442
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1443 if (!PyArg_ParseTupleAndKeywords(args, kwds, "O\|s:StreamReader",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1444 streamkwarglist, &stream, &errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1445 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1446
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1447 self = (MultibyteStreamReaderObject *)type->tp_alloc(type, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1448 if (self == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1449 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1450
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1451 codec = PyObject_GetAttrString((PyObject *)type, "codec");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1452 if (codec == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1453 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1454 if (!MultibyteCodec_Check(codec)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1455 PyErr_SetString(PyExc_TypeError, "codec is unexpected type");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1456 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1457 }
37233 0098e7264eaa Fix reference leaks introduced by the recent incremental codec Hye-Shik Chang <hyeshik@gmail.com> parents: 37224 diff changeset	1458
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1459 self->codec = ((MultibyteCodecObject *)codec)->codec;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1460 self->stream = stream;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1461 Py_INCREF(stream);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1462 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1463 self->errors = internal_error_callback(errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1464 if (self->errors == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1465 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1466 if (self->codec->decinit != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1467 self->codec->decinit(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1468 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1469
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1470 Py_DECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1471 return (PyObject *)self;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1472
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1473 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1474 Py_XDECREF(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1475 Py_XDECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1476 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1477 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1478
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1479 static int
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1480 mbstreamreader_init(PyObject self, PyObject args, PyObject *kwds)
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1481 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1482 return 0;
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1483 }
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1484
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1485 static int
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1486 mbstreamreader_traverse(MultibyteStreamReaderObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1487 visitproc visit, void *arg)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1488 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1489 if (ERROR_ISCUSTOM(self->errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1490 Py_VISIT(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1491 Py_VISIT(self->stream);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1492 return 0;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1493 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1494
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1495 static void
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1496 mbstreamreader_dealloc(MultibyteStreamReaderObject *self)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1497 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1498 PyObject_GC_UnTrack(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1499 ERROR_DECREF(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1500 Py_XDECREF(self->stream);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1501 Py_TYPE(self)->tp_free(self);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1502 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1503
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1504 static PyTypeObject MultibyteStreamReader_Type = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1505 PyVarObject_HEAD_INIT(NULL, 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1506 "MultibyteStreamReader", /* tp_name */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1507 sizeof(MultibyteStreamReaderObject), /* tp_basicsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1508 0, /* tp_itemsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1509 /* methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1510 (destructor)mbstreamreader_dealloc, /* tp_dealloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1511 0, /* tp_print */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1512 0, /* tp_getattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1513 0, /* tp_setattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1514 0, /* tp_compare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1515 0, /* tp_repr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1516 0, /* tp_as_number */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1517 0, /* tp_as_sequence */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1518 0, /* tp_as_mapping */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1519 0, /* tp_hash */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1520 0, /* tp_call */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1521 0, /* tp_str */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1522 PyObject_GenericGetAttr, /* tp_getattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1523 0, /* tp_setattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1524 0, /* tp_as_buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1525 Py_TPFLAGS_DEFAULT \| Py_TPFLAGS_HAVE_GC
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1526 \| Py_TPFLAGS_BASETYPE, /* tp_flags */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1527 0, /* tp_doc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1528 (traverseproc)mbstreamreader_traverse, /* tp_traverse */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1529 0, /* tp_clear */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1530 0, /* tp_richcompare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1531 0, /* tp_weaklistoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1532 0, /* tp_iter */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1533 0, /* tp_iterext */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1534 mbstreamreader_methods, /* tp_methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1535 mbstreamreader_members, /* tp_members */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1536 codecctx_getsets, /* tp_getset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1537 0, /* tp_base */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1538 0, /* tp_dict */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1539 0, /* tp_descr_get */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1540 0, /* tp_descr_set */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1541 0, /* tp_dictoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1542 mbstreamreader_init, /* tp_init */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1543 0, /* tp_alloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1544 mbstreamreader_new, /* tp_new */
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1545 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1546
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1547
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1548 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1549 * MultibyteStreamWriter object
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1550 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1551
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1552 static int
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1553 mbstreamwriter_iwrite(MultibyteStreamWriterObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1554 PyObject *unistr)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1555 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1556 PyObject str, wr;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1557
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1558 str = encoder_encode_stateful(STATEFUL_ECTX(self), unistr, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1559 if (str == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1560 return -1;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1561
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1562 wr = PyObject_CallMethod(self->stream, "write", "O", str);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1563 Py_DECREF(str);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1564 if (wr == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1565 return -1;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1566
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1567 Py_DECREF(wr);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1568 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1569 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1570
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1571 static PyObject *
38489 3fe3ccdf0155 Make use of METH_O and METH_NOARGS where possible. Georg Brandl <georg@python.org> parents: 37919 diff changeset	1572 mbstreamwriter_write(MultibyteStreamWriterObject self, PyObject strobj)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1573 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1574 if (mbstreamwriter_iwrite(self, strobj))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1575 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1576 else
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1577 Py_RETURN_NONE;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1578 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1579
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1580 static PyObject *
38489 3fe3ccdf0155 Make use of METH_O and METH_NOARGS where possible. Georg Brandl <georg@python.org> parents: 37919 diff changeset	1581 mbstreamwriter_writelines(MultibyteStreamWriterObject self, PyObject lines)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1582 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1583 PyObject *strobj;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1584 int i, r;
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	1585
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1586 if (!PySequence_Check(lines)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1587 PyErr_SetString(PyExc_TypeError,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1588 "arg must be a sequence object");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1589 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1590 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1591
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1592 for (i = 0; i < PySequence_Length(lines); i++) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1593 /* length can be changed even within this loop */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1594 strobj = PySequence_GetItem(lines, i);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1595 if (strobj == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1596 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1597
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1598 r = mbstreamwriter_iwrite(self, strobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1599 Py_DECREF(strobj);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1600 if (r == -1)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1601 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1602 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1603
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1604 Py_RETURN_NONE;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1605 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1606
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1607 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1608 mbstreamwriter_reset(MultibyteStreamWriterObject *self)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1609 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1610 const Py_UNICODE *pending;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1611 PyObject *pwrt;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1612
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1613 pending = self->pending;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1614 pwrt = multibytecodec_encode(self->codec, &self->state,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1615 &pending, self->pendingsize, self->errors,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1616 MBENC_FLUSH \| MBENC_RESET);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1617 /* some pending buffer can be truncated when UnicodeEncodeError is
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1618 * raised on 'strict' mode. but, 'reset' method is designed to
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1619 * reset the pending buffer or states so failed string sequence
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1620 * ought to be missed */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1621 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1622 if (pwrt == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1623 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1624
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1625 if (PyString_Size(pwrt) > 0) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1626 PyObject *wr;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1627 wr = PyObject_CallMethod(self->stream, "write", "O", pwrt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1628 if (wr == NULL) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1629 Py_DECREF(pwrt);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1630 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1631 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1632 }
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1633 Py_DECREF(pwrt);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1634
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1635 Py_RETURN_NONE;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1636 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1637
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1638 static PyObject *
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1639 mbstreamwriter_new(PyTypeObject type, PyObject args, PyObject *kwds)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1640 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1641 MultibyteStreamWriterObject *self;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1642 PyObject stream, codec = NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1643 char *errors = NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1644
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1645 if (!PyArg_ParseTupleAndKeywords(args, kwds, "O\|s:StreamWriter",
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1646 streamkwarglist, &stream, &errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1647 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1648
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1649 self = (MultibyteStreamWriterObject *)type->tp_alloc(type, 0);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1650 if (self == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1651 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1652
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1653 codec = PyObject_GetAttrString((PyObject *)type, "codec");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1654 if (codec == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1655 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1656 if (!MultibyteCodec_Check(codec)) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1657 PyErr_SetString(PyExc_TypeError, "codec is unexpected type");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1658 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1659 }
37233 0098e7264eaa Fix reference leaks introduced by the recent incremental codec Hye-Shik Chang <hyeshik@gmail.com> parents: 37224 diff changeset	1660
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1661 self->codec = ((MultibyteCodecObject *)codec)->codec;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1662 self->stream = stream;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1663 Py_INCREF(stream);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1664 self->pendingsize = 0;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1665 self->errors = internal_error_callback(errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1666 if (self->errors == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1667 goto errorexit;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1668 if (self->codec->encinit != NULL &&
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1669 self->codec->encinit(&self->state, self->codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1670 goto errorexit;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1671
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1672 Py_DECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1673 return (PyObject *)self;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1674
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1675 errorexit:
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1676 Py_XDECREF(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1677 Py_XDECREF(codec);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1678 return NULL;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1679 }
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1680
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1681 static int
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1682 mbstreamwriter_init(PyObject self, PyObject args, PyObject *kwds)
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1683 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1684 return 0;
37901 fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1685 }
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1686
fd6f190e26a8 Backport p3yk r45619: Hye-Shik Chang <hyeshik@gmail.com> parents: 37262 diff changeset	1687 static int
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1688 mbstreamwriter_traverse(MultibyteStreamWriterObject *self,
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1689 visitproc visit, void *arg)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1690 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1691 if (ERROR_ISCUSTOM(self->errors))
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1692 Py_VISIT(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1693 Py_VISIT(self->stream);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1694 return 0;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1695 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1696
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1697 static void
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1698 mbstreamwriter_dealloc(MultibyteStreamWriterObject *self)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1699 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1700 PyObject_GC_UnTrack(self);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1701 ERROR_DECREF(self->errors);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1702 Py_XDECREF(self->stream);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1703 Py_TYPE(self)->tp_free(self);
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1704 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1705
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1706 static struct PyMethodDef mbstreamwriter_methods[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1707 {"write", (PyCFunction)mbstreamwriter_write,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1708 METH_O, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1709 {"writelines", (PyCFunction)mbstreamwriter_writelines,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1710 METH_O, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1711 {"reset", (PyCFunction)mbstreamwriter_reset,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1712 METH_NOARGS, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1713 {NULL, NULL},
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1714 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1715
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1716 static PyMemberDef mbstreamwriter_members[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1717 {"stream", T_OBJECT,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1718 offsetof(MultibyteStreamWriterObject, stream),
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1719 READONLY, NULL},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1720 {NULL,}
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1721 };
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1722
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1723 static PyTypeObject MultibyteStreamWriter_Type = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1724 PyVarObject_HEAD_INIT(NULL, 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1725 "MultibyteStreamWriter", /* tp_name */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1726 sizeof(MultibyteStreamWriterObject), /* tp_basicsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1727 0, /* tp_itemsize */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1728 /* methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1729 (destructor)mbstreamwriter_dealloc, /* tp_dealloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1730 0, /* tp_print */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1731 0, /* tp_getattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1732 0, /* tp_setattr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1733 0, /* tp_compare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1734 0, /* tp_repr */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1735 0, /* tp_as_number */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1736 0, /* tp_as_sequence */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1737 0, /* tp_as_mapping */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1738 0, /* tp_hash */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1739 0, /* tp_call */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1740 0, /* tp_str */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1741 PyObject_GenericGetAttr, /* tp_getattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1742 0, /* tp_setattro */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1743 0, /* tp_as_buffer */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1744 Py_TPFLAGS_DEFAULT \| Py_TPFLAGS_HAVE_GC
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1745 \| Py_TPFLAGS_BASETYPE, /* tp_flags */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1746 0, /* tp_doc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1747 (traverseproc)mbstreamwriter_traverse, /* tp_traverse */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1748 0, /* tp_clear */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1749 0, /* tp_richcompare */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1750 0, /* tp_weaklistoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1751 0, /* tp_iter */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1752 0, /* tp_iterext */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1753 mbstreamwriter_methods, /* tp_methods */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1754 mbstreamwriter_members, /* tp_members */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1755 codecctx_getsets, /* tp_getset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1756 0, /* tp_base */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1757 0, /* tp_dict */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1758 0, /* tp_descr_get */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1759 0, /* tp_descr_set */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1760 0, /* tp_dictoffset */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1761 mbstreamwriter_init, /* tp_init */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1762 0, /* tp_alloc */
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1763 mbstreamwriter_new, /* tp_new */
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1764 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1765
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1766
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1767 /**
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1768 * Exposed factory function
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1769 */
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1770
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1771 static PyObject *
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1772 __create_codec(PyObject ignore, PyObject arg)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1773 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1774 MultibyteCodecObject *self;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1775 MultibyteCodec *codec;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1776
61463 fd25030223b0 Untabify C files. Will watch buildbots. Antoine Pitrou <solipsis@pitrou.net> parents: 59998 diff changeset	1777 if (!PyCapsule_IsValid(arg, PyMultibyteCodec_CAPSULE_NAME)) {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1778 PyErr_SetString(PyExc_ValueError, "argument type invalid");
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1779 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1780 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1781
61463 fd25030223b0 Untabify C files. Will watch buildbots. Antoine Pitrou <solipsis@pitrou.net> parents: 59998 diff changeset	1782 codec = PyCapsule_GetPointer(arg, PyMultibyteCodec_CAPSULE_NAME);
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1783 if (codec->codecinit != NULL && codec->codecinit(codec->config) != 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1784 return NULL;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1785
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1786 self = PyObject_New(MultibyteCodecObject, &MultibyteCodec_Type);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1787 if (self == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1788 return NULL;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1789 self->codec = codec;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1790
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1791 return (PyObject *)self;
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1792 }
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1793
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1794 static struct PyMethodDef __methods[] = {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1795 {"__create_codec", (PyCFunction)__create_codec, METH_O},
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1796 {NULL, NULL},
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1797 };
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1798
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1799 PyMODINIT_FUNC
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1800 init_multibytecodec(void)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1801 {
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1802 int i;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1803 PyObject *m;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1804 PyTypeObject *typelist[] = {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1805 &MultibyteIncrementalEncoder_Type,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1806 &MultibyteIncrementalDecoder_Type,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1807 &MultibyteStreamReader_Type,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1808 &MultibyteStreamWriter_Type,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1809 NULL
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1810 };
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1811
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1812 if (PyType_Ready(&MultibyteCodec_Type) < 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1813 return;
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 36831 diff changeset	1814
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1815 m = Py_InitModule("_multibytecodec", __methods);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1816 if (m == NULL)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1817 return;
35627 4e0d2a7f5f15 Backport fix for patch #1297028, cjkcodecs does not initialize type pointer Neal Norwitz <nnorwitz@gmail.com> parents: 33193 diff changeset	1818
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1819 for (i = 0; typelist[i] != NULL; i++) {
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1820 if (PyType_Ready(typelist[i]) < 0)
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1821 return;
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1822 Py_INCREF(typelist[i]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1823 PyModule_AddObject(m, typelist[i]->tp_name,
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1824 (PyObject *)typelist[i]);
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1825 }
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1826
61465 e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1827 if (PyErr_Occurred())
e38335d4df91 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 49431 diff changeset	1828 Py_FatalError("can't initialize the _multibytecodec module");
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1829 }

Mercurial > cpython-lru_cache-weakref

annotate Modules/cjkcodecs/multibytecodec.c @ 79675:c0a2294de787 2.7