Mercurial > cpython-lru_cache-weakref

annotate Parser/tokenizer.c @ 58664:993f703a61ba

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merged revisions 77484,77487,77561,77570,77593,77603,77608,77667,77702-77703,77739,77858,77887,77889 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r77484 | skip.montanaro | 2010-01-13 19:12:34 -0600 (Wed, 13 Jan 2010) | 4 lines Update PyEval_EvalFrame to PyEval_EvalFrameEx. This looks to have been done partially before. Also add a comment describing how this might have to work with different versions of the interpreter. ........ r77487 | ezio.melotti | 2010-01-14 05:34:10 -0600 (Thu, 14 Jan 2010) | 1 line Fixed typo ........ r77561 | georg.brandl | 2010-01-17 02:42:30 -0600 (Sun, 17 Jan 2010) | 1 line #7699: improve datetime docs: straightforward linking to strftime/strptime section, mark classmethods as such. ........ r77570 | georg.brandl | 2010-01-17 06:14:42 -0600 (Sun, 17 Jan 2010) | 1 line Add note about usage of STRINGLIB_EMPTY. ........ r77593 | georg.brandl | 2010-01-17 17:33:53 -0600 (Sun, 17 Jan 2010) | 1 line Fix internal reference. ........ r77603 | benjamin.peterson | 2010-01-18 17:07:56 -0600 (Mon, 18 Jan 2010) | 8 lines data descriptors do not override the class dictionary if __get__ is not defined Adjust documentation and add a test to verify this behavior. See http://mail.python.org/pipermail/python-dev/2010-January/095637.html for discussion. ........ r77608 | gregory.p.smith | 2010-01-19 02:19:03 -0600 (Tue, 19 Jan 2010) | 6 lines Do not compile stubs for the sha2 series hashes in the openssl hashlib module when the openssl version is too old to support them. That leads both compiled code bloat and to unittests attempting to test implementations that don't exist for comparison purposes on such platforms. ........ r77667 | mark.dickinson | 2010-01-21 12:32:27 -0600 (Thu, 21 Jan 2010) | 1 line Add two more test_strtod test values. ........ r77702 | georg.brandl | 2010-01-23 02:43:31 -0600 (Sat, 23 Jan 2010) | 1 line #7762: fix refcount annotation of PyUnicode_Tailmatch(). ........ r77703 | georg.brandl | 2010-01-23 02:47:54 -0600 (Sat, 23 Jan 2010) | 1 line #7725: fix referencing issue. ........ r77739 | benjamin.peterson | 2010-01-24 21:52:52 -0600 (Sun, 24 Jan 2010) | 1 line mention from_float() in error message ........ r77858 | georg.brandl | 2010-01-30 11:57:48 -0600 (Sat, 30 Jan 2010) | 1 line #7802: fix invalid example (heh). ........ r77887 | georg.brandl | 2010-01-31 12:51:49 -0600 (Sun, 31 Jan 2010) | 5 lines Fix-up ftplib documentation: move exception descriptions to toplevel, not inside a class remove attribution in "versionadded" spell and grammar check docstring of FTP_TLS ........ r77889 | michael.foord | 2010-01-31 13:59:26 -0600 (Sun, 31 Jan 2010) | 1 line Minor modification to unittest documentation. ........
author Benjamin Peterson <benjamin@python.org>
date Wed, 03 Feb 2010 02:35:45 +0000
parents a769a9b34fbc
children d4a188dac222
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
145
853b736a6a9e Added copyright notice.
Guido van Rossum <guido@python.org>
parents: 96
diff changeset
1
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
2 /* Tokenizer implementation */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
3
23029
8291473fde3e Mass checkin of universal newline support.
Jack Jansen <jack.jansen@cwi.nl>
parents: 19621
diff changeset
4 #include "Python.h"
86
a6934380c6e7 "Compiling" version
Guido van Rossum <guido@python.org>
parents: 67
diff changeset
5 #include "pgenheaders.h"
a6934380c6e7 "Compiling" version
Guido van Rossum <guido@python.org>
parents: 67
diff changeset
6
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
7 #include <ctype.h>
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
8 #include <assert.h>
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
9
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
10 #include "tokenizer.h"
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 #include "errcode.h"
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
12
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
13 #ifndef PGEN
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
14 #include "unicodeobject.h"
47302
811a80651232 Renamed files bytesobject.[ch] and stringobject.[ch]
Christian Heimes <christian@cheimes.de>
parents: 47301
diff changeset
15 #include "bytesobject.h"
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
16 #include "fileobject.h"
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
17 #include "codecs.h"
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
18 #include "abstract.h"
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
19 #endif /* PGEN */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
20
42013
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
21 #define is_potential_identifier_start(c) (\
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
22 (c >= 'a' && c <= 'z')\
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
23 || (c >= 'A' && c <= 'Z')\
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
24 || c == '_'\
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
25 || (c >= 128))
42013
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
26
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
27 #define is_potential_identifier_char(c) (\
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
28 (c >= 'a' && c <= 'z')\
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
29 || (c >= 'A' && c <= 'Z')\
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
30 || (c >= '0' && c <= '9')\
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
31 || c == '_'\
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
32 || (c >= 128))
42013
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
33
25924
dc4a0336a2a3 Patch #512981: Update readline input stream on sys.stdin/out change.
Martin v. Löwis <martin@v.loewis.de>
parents: 25101
diff changeset
34 extern char *PyOS_Readline(FILE *, FILE *, char *);
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
35 /* Return malloc'ed string including trailing \n;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
36 empty malloc'ed string for EOF;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
37 NULL if interrupted */
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
38
678
205b947b1d3c Make tabs always 8 spaces wide -- it's more portable.
Guido van Rossum <guido@python.org>
parents: 477
diff changeset
39 /* Don't ever change this -- it would break the portability of Python code */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
40 #define TABSIZE 8
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
41
86
a6934380c6e7 "Compiling" version
Guido van Rossum <guido@python.org>
parents: 67
diff changeset
42 /* Forward */
13447
a89f7f268de4 Nuke all remaining occurrences of Py_PROTO and Py_FPROTO.
Tim Peters <tim.peters@gmail.com>
parents: 13256
diff changeset
43 static struct tok_state *tok_new(void);
a89f7f268de4 Nuke all remaining occurrences of Py_PROTO and Py_FPROTO.
Tim Peters <tim.peters@gmail.com>
parents: 13256
diff changeset
44 static int tok_nextc(struct tok_state *tok);
a89f7f268de4 Nuke all remaining occurrences of Py_PROTO and Py_FPROTO.
Tim Peters <tim.peters@gmail.com>
parents: 13256
diff changeset
45 static void tok_backup(struct tok_state *tok, int c);
86
a6934380c6e7 "Compiling" version
Guido van Rossum <guido@python.org>
parents: 67
diff changeset
46
43607
7593436058dd Plug a memory leak where a struct tok_state was not being freed.
Brett Cannon <bcannon@gmail.com>
parents: 43605
diff changeset
47
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
48 /* Token names */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
49
5379
1c9a84e0981d Another directory quickly renamed.
Guido van Rossum <guido@python.org>
parents: 5341
diff changeset
50 char *_PyParser_TokenNames[] = {
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
51 "ENDMARKER",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
52 "NAME",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 "NUMBER",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 "STRING",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 "NEWLINE",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
56 "INDENT",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
57 "DEDENT",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 "LPAR",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 "RPAR",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
60 "LSQB",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
61 "RSQB",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
62 "COLON",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
63 "COMMA",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
64 "SEMI",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
65 "PLUS",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
66 "MINUS",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
67 "STAR",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
68 "SLASH",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
69 "VBAR",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
70 "AMPER",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
71 "LESS",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
72 "GREATER",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
73 "EQUAL",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
74 "DOT",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
75 "PERCENT",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
76 "LBRACE",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
77 "RBRACE",
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
78 "EQEQUAL",
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
79 "NOTEQUAL",
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
80 "LESSEQUAL",
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
81 "GREATEREQUAL",
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
82 "TILDE",
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
83 "CIRCUMFLEX",
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
84 "LEFTSHIFT",
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
85 "RIGHTSHIFT",
3333
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
86 "DOUBLESTAR",
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
87 "PLUSEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
88 "MINEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
89 "STAREQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
90 "SLASHEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
91 "PERCENTEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
92 "AMPEREQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
93 "VBAREQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
94 "CIRCUMFLEXEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
95 "LEFTSHIFTEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
96 "RIGHTSHIFTEQUAL",
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
97 "DOUBLESTAREQUAL",
19051
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
98 "DOUBLESLASH",
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
99 "DOUBLESLASHEQUAL",
32888
c7e03cf8333e PEP-0318, @decorator-style. In Guido's words:
Anthony Baxter <anthonybaxter@gmail.com>
parents: 30979
diff changeset
100 "AT",
40529
54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds
Neal Norwitz <nnorwitz@gmail.com>
parents: 40450
diff changeset
101 "RARROW",
41230
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
102 "ELLIPSIS",
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
103 /* This table must match the #defines in token.h! */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
104 "OP",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
105 "<ERRORTOKEN>",
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
106 "<N_TOKENS>"
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
107 };
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
108
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
109
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
110 /* Create and initialize a new tok_state structure */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
111
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
112 static struct tok_state *
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
113 tok_new(void)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
114 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
115 struct tok_state *tok = (struct tok_state *)PyMem_MALLOC(
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
116 sizeof(struct tok_state));
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
117 if (tok == NULL)
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
118 return NULL;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
119 tok->buf = tok->cur = tok->end = tok->inp = tok->start = NULL;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
120 tok->done = E_OK;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 tok->fp = NULL;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
122 tok->input = NULL;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 tok->tabsize = TABSIZE;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 tok->indent = 0;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 tok->indstack[0] = 0;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 tok->atbol = 1;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
127 tok->pendin = 0;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
128 tok->prompt = tok->nextprompt = NULL;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 tok->lineno = 0;
1284
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
130 tok->level = 0;
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
131 tok->filename = NULL;
37704
bb3f9076d079 Make 'python -tt' the default, meaning Python won't allow mixing tabs and
Thomas Wouters <thomas@python.org>
parents: 36809
diff changeset
132 tok->altwarning = 1;
bb3f9076d079 Make 'python -tt' the default, meaning Python won't allow mixing tabs and
Thomas Wouters <thomas@python.org>
parents: 36809
diff changeset
133 tok->alterror = 1;
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
134 tok->alttabsize = 1;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
135 tok->altindstack[0] = 0;
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
136 tok->decoding_state = STATE_INIT;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
137 tok->decoding_erred = 0;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
138 tok->read_coding_spec = 0;
49539
481ad3d9ffd1 Latin-1 source code was not being properly decoded when passed through
Brett Cannon <bcannon@gmail.com>
parents: 49484
diff changeset
139 tok->enc = NULL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
140 tok->encoding = NULL;
25098
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
141 tok->cont_line = 0;
24544
0bd521eb714e Make pgen compile with pydebug. Duplicate normalized names, as it may
Martin v. Löwis <martin@v.loewis.de>
parents: 24543
diff changeset
142 #ifndef PGEN
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
143 tok->decoding_readline = NULL;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
144 tok->decoding_buffer = NULL;
24544
0bd521eb714e Make pgen compile with pydebug. Duplicate normalized names, as it may
Martin v. Löwis <martin@v.loewis.de>
parents: 24543
diff changeset
145 #endif
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
146 return tok;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
148
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
149 static char *
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
150 new_string(const char *s, Py_ssize_t len)
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
151 {
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
152 char* result = (char *)PyMem_MALLOC(len + 1);
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
153 if (result != NULL) {
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
154 memcpy(result, s, len);
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
155 result[len] = '\0';
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
156 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
157 return result;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
158 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
159
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
160 #ifdef PGEN
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
161
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
162 static char *
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
163 decoding_fgets(char *s, int size, struct tok_state *tok)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
164 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
165 return fgets(s, size, tok->fp);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
166 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
167
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
168 static int
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
169 decoding_feof(struct tok_state *tok)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
170 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
171 return feof(tok->fp);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
172 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
173
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
174 static char *
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
175 decode_str(const char *str, int exec_input, struct tok_state *tok)
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
176 {
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
177 return new_string(str, strlen(str));
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
178 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
179
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
180 #else /* PGEN */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
181
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
182 static char *
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
183 error_ret(struct tok_state *tok) /* XXX */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
184 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
185 tok->decoding_erred = 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
186 if (tok->fp != NULL && tok->buf != NULL) /* see PyTokenizer_Free */
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
187 PyMem_FREE(tok->buf);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
188 tok->buf = NULL;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
189 return NULL; /* as if it were EOF */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
190 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
191
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
192
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
193 static char *
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
194 get_normal_name(char *s) /* for utf-8 and latin-1 */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
195 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
196 char buf[13];
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
197 int i;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
198 for (i = 0; i < 12; i++) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
199 int c = s[i];
57209
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
200 if (c == '\0')
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
201 break;
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
202 else if (c == '_')
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
203 buf[i] = '-';
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
204 else
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
205 buf[i] = tolower(c);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
206 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
207 buf[i] = '\0';
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
208 if (strcmp(buf, "utf-8") == 0 ||
57209
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
209 strncmp(buf, "utf-8-", 6) == 0)
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
210 return "utf-8";
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
211 else if (strcmp(buf, "latin-1") == 0 ||
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
212 strcmp(buf, "iso-8859-1") == 0 ||
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
213 strcmp(buf, "iso-latin-1") == 0 ||
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
214 strncmp(buf, "latin-1-", 8) == 0 ||
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
215 strncmp(buf, "iso-8859-1-", 11) == 0 ||
57209
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
216 strncmp(buf, "iso-latin-1-", 12) == 0)
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
217 return "iso-8859-1";
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
218 else
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
219 return s;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
220 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
221
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
222 /* Return the coding spec in S, or NULL if none is found. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
223
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
224 static char *
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
225 get_coding_spec(const char *s, Py_ssize_t size)
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
226 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
227 Py_ssize_t i;
25098
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
228 /* Coding spec must be in a comment, and that comment must be
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
229 * the only statement on the source code line. */
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
230 for (i = 0; i < size - 6; i++) {
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
231 if (s[i] == '#')
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
232 break;
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
233 if (s[i] != ' ' && s[i] != '\t' && s[i] != '\014')
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
234 return NULL;
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
235 }
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
236 for (; i < size - 6; i++) { /* XXX inefficient search */
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
237 const char* t = s + i;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
238 if (strncmp(t, "coding", 6) == 0) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
239 const char* begin = NULL;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
240 t += 6;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
241 if (t[0] != ':' && t[0] != '=')
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
242 continue;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
243 do {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
244 t++;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
245 } while (t[0] == '\x20' || t[0] == '\t');
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
246
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
247 begin = t;
36090
8cb686dd6e64 Fix SF bug #1072182, problems with signed characters.
Neal Norwitz <nnorwitz@gmail.com>
parents: 36078
diff changeset
248 while (isalnum(Py_CHARMASK(t[0])) ||
25995
1aaa12c08b35 Fix compiler warning on HP-UX.
Neal Norwitz <nnorwitz@gmail.com>
parents: 25924
diff changeset
249 t[0] == '-' || t[0] == '_' || t[0] == '.')
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
250 t++;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
251
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
252 if (begin < t) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
253 char* r = new_string(begin, t - begin);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
254 char* q = get_normal_name(r);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
255 if (r != q) {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
256 PyMem_FREE(r);
24544
0bd521eb714e Make pgen compile with pydebug. Duplicate normalized names, as it may
Martin v. Löwis <martin@v.loewis.de>
parents: 24543
diff changeset
257 r = new_string(q, strlen(q));
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
258 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
259 return r;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
260 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
261 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
262 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
263 return NULL;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
264 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
265
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
266 /* Check whether the line contains a coding spec. If it does,
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
267 invoke the set_readline function for the new encoding.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
268 This function receives the tok_state and the new encoding.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
269 Return 1 on success, 0 on failure. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
270
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
271 static int
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
272 check_coding_spec(const char* line, Py_ssize_t size, struct tok_state *tok,
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
273 int set_readline(struct tok_state *, const char *))
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
274 {
25101
32331d26d7cf Removed reliance on gcc/C99 extension.
Tim Peters <tim.peters@gmail.com>
parents: 25098
diff changeset
275 char * cs;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
276 int r = 1;
25101
32331d26d7cf Removed reliance on gcc/C99 extension.
Tim Peters <tim.peters@gmail.com>
parents: 25098
diff changeset
277
25098
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
278 if (tok->cont_line)
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
279 /* It's a continuation line, so it can't be a coding spec. */
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
280 return 1;
25101
32331d26d7cf Removed reliance on gcc/C99 extension.
Tim Peters <tim.peters@gmail.com>
parents: 25098
diff changeset
281 cs = get_coding_spec(line, size);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
282 if (cs != NULL) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
283 tok->read_coding_spec = 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
284 if (tok->encoding == NULL) {
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
285 assert(tok->decoding_state == STATE_RAW);
49539
481ad3d9ffd1 Latin-1 source code was not being properly decoded when passed through
Brett Cannon <bcannon@gmail.com>
parents: 49484
diff changeset
286 if (strcmp(cs, "utf-8") == 0) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
287 tok->encoding = cs;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
288 } else {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
289 r = set_readline(tok, cs);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
290 if (r) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
291 tok->encoding = cs;
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
292 tok->decoding_state = STATE_NORMAL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
293 }
35751
e56144704808 Free coding spec (cs) if there was an error to prevent mem leak. Maybe backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 35666
diff changeset
294 else
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
295 PyMem_FREE(cs);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
296 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
297 } else { /* then, compare cs with BOM */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
298 r = (strcmp(tok->encoding, cs) == 0);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
299 PyMem_FREE(cs);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
300 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
301 }
36078
1ff4790b7605 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35878
diff changeset
302 if (!r) {
1ff4790b7605 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35878
diff changeset
303 cs = tok->encoding;
1ff4790b7605 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35878
diff changeset
304 if (!cs)
1ff4790b7605 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35878
diff changeset
305 cs = "with BOM";
1ff4790b7605 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35878
diff changeset
306 PyErr_Format(PyExc_SyntaxError, "encoding problem: %s", cs);
1ff4790b7605 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35878
diff changeset
307 }
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
308 return r;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
309 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
310
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
311 /* See whether the file starts with a BOM. If it does,
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
312 invoke the set_readline function with the new encoding.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
313 Return 1 on success, 0 on failure. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
314
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
315 static int
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
316 check_bom(int get_char(struct tok_state *),
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
317 void unget_char(int, struct tok_state *),
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
318 int set_readline(struct tok_state *, const char *),
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
319 struct tok_state *tok)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
320 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
321 int ch = get_char(tok);
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
322 tok->decoding_state = STATE_RAW;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
323 if (ch == EOF) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
324 return 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
325 } else if (ch == 0xEF) {
43974
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
326 ch = get_char(tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
327 if (ch != 0xBB) {
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
328 unget_char(ch, tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
329 unget_char(0xEF, tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
330 /* any token beginning with '\xEF' is a bad token */
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
331 return 1;
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
332 }
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
333 ch = get_char(tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
334 if (ch != 0xBF) {
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
335 unget_char(ch, tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
336 unget_char(0xBB, tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
337 unget_char(0xEF, tok);
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
338 /* any token beginning with '\xEF' is a bad token */
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
339 return 1;
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
340 }
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
341 #if 0
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
342 /* Disable support for UTF-16 BOMs until a decision
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
343 is made whether this needs to be supported. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
344 } else if (ch == 0xFE) {
57209
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
345 ch = get_char(tok);
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
346 if (ch != 0xFF)
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
347 goto NON_BOM;
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
348 if (!set_readline(tok, "utf-16-be"))
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
349 return 0;
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
350 tok->decoding_state = STATE_NORMAL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
351 } else if (ch == 0xFF) {
57209
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
352 ch = get_char(tok);
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
353 if (ch != 0xFE)
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
354 goto NON_BOM;
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
355 if (!set_readline(tok, "utf-16-le"))
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
356 return 0;
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
357 tok->decoding_state = STATE_NORMAL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
358 #endif
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
359 } else {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
360 unget_char(ch, tok);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
361 return 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
362 }
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
363 if (tok->encoding != NULL)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
364 PyMem_FREE(tok->encoding);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
365 tok->encoding = new_string("utf-8", 5); /* resulting is in utf-8 */
43974
da478471f1b6 Problem found while converting from PyBytes to PyString:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43879
diff changeset
366 /* No need to set_readline: input is already utf-8 */
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
367 return 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
368 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
369
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
370 /* Read a line of text from TOK into S, using the stream in TOK.
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
371 Return NULL on failure, else S.
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
372
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
373 On entry, tok->decoding_buffer will be one of:
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
374 1) NULL: need to call tok->decoding_readline to get a new line
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
375 2) PyUnicodeObject *: decoding_feof has called tok->decoding_readline and
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
376 stored the result in tok->decoding_buffer
47300
96b34263e448 Renamed PyBytes to PyByteArray
Christian Heimes <christian@cheimes.de>
parents: 46664
diff changeset
377 3) PyByteArrayObject *: previous call to fp_readl did not have enough room
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
378 (in the s buffer) to copy entire contents of the line read
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
379 by tok->decoding_readline. tok->decoding_buffer has the overflow.
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
380 In this case, fp_readl is called in a loop (with an expanded buffer)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
381 until the buffer ends with a '\n' (or until the end of the file is
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
382 reached): see tok_nextc and its calls to decoding_fgets.
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
383 */
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
384
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
385 static char *
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
386 fp_readl(char *s, int size, struct tok_state *tok)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
387 {
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
388 PyObject* bufobj;
41670
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
389 const char *buf;
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
390 Py_ssize_t buflen;
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
391
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
392 /* Ask for one less byte so we can terminate it */
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
393 assert(size > 0);
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
394 size--;
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
395
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
396 if (tok->decoding_buffer) {
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
397 bufobj = tok->decoding_buffer;
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
398 Py_INCREF(bufobj);
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
399 }
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
400 else
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
401 {
41670
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
402 bufobj = PyObject_CallObject(tok->decoding_readline, NULL);
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
403 if (bufobj == NULL)
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
404 goto error;
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
405 }
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
406 if (PyUnicode_CheckExact(bufobj))
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
407 {
48660
85b67fb5672b Rename PyUnicode_AsString -> _PyUnicode_AsString and
Marc-André Lemburg <mal@egenix.com>
parents: 47302
diff changeset
408 buf = _PyUnicode_AsStringAndSize(bufobj, &buflen);
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
409 if (buf == NULL) {
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
410 goto error;
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
411 }
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
412 }
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
413 else
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
414 {
47300
96b34263e448 Renamed PyBytes to PyByteArray
Christian Heimes <christian@cheimes.de>
parents: 46664
diff changeset
415 buf = PyByteArray_AsString(bufobj);
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
416 if (buf == NULL) {
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
417 goto error;
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
418 }
47300
96b34263e448 Renamed PyBytes to PyByteArray
Christian Heimes <christian@cheimes.de>
parents: 46664
diff changeset
419 buflen = PyByteArray_GET_SIZE(bufobj);
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
420 }
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
421
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
422 Py_XDECREF(tok->decoding_buffer);
41670
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
423 if (buflen > size) {
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
424 /* Too many chars, the rest goes into tok->decoding_buffer */
47300
96b34263e448 Renamed PyBytes to PyByteArray
Christian Heimes <christian@cheimes.de>
parents: 46664
diff changeset
425 tok->decoding_buffer = PyByteArray_FromStringAndSize(buf+size,
41670
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
426 buflen-size);
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
427 if (tok->decoding_buffer == NULL)
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
428 goto error;
41670
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
429 buflen = size;
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
430 }
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
431 else
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
432 tok->decoding_buffer = NULL;
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
433
41670
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
434 memcpy(s, buf, buflen);
688e2bc20802 Use AsCharBuffer to get a C string out of a Python string.
Guido van Rossum <guido@python.org>
parents: 41647
diff changeset
435 s[buflen] = '\0';
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
436 if (buflen == 0) /* EOF */
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
437 s = NULL;
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
438 Py_DECREF(bufobj);
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
439 return s;
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
440
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
441 error:
43873
1739869cbc6e Correction for issue1134: all source files with a coding spec, except latin-1
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 43777
diff changeset
442 Py_XDECREF(bufobj);
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
443 return error_ret(tok);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
444 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
445
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
446 /* Set the readline function for TOK to a StreamReader's
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
447 readline function. The StreamReader is named ENC.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
448
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
449 This function is called from check_bom and check_coding_spec.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
450
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
451 ENC is usually identical to the future value of tok->encoding,
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
452 except for the (currently unsupported) case of UTF-16.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
453
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
454 Return 1 on success, 0 on failure. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
455
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
456 static int
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
457 fp_setreadl(struct tok_state *tok, const char* enc)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
458 {
41970
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
459 PyObject *readline = NULL, *stream = NULL, *io = NULL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
460
44442
fa2f7183b1af More PyImport_ImportModule -> PyImport_ImportModuleNoBlock
Christian Heimes <christian@cheimes.de>
parents: 44351
diff changeset
461 io = PyImport_ImportModuleNoBlock("io");
41970
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
462 if (io == NULL)
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
463 goto cleanup;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
464
49007
45c43808979e PyTokenizer_FindEncoding() always failed because it set the tokenizer state
Brett Cannon <bcannon@gmail.com>
parents: 48660
diff changeset
465 if (tok->filename)
45c43808979e PyTokenizer_FindEncoding() always failed because it set the tokenizer state
Brett Cannon <bcannon@gmail.com>
parents: 48660
diff changeset
466 stream = PyObject_CallMethod(io, "open", "ssis",
45c43808979e PyTokenizer_FindEncoding() always failed because it set the tokenizer state
Brett Cannon <bcannon@gmail.com>
parents: 48660
diff changeset
467 tok->filename, "r", -1, enc);
45c43808979e PyTokenizer_FindEncoding() always failed because it set the tokenizer state
Brett Cannon <bcannon@gmail.com>
parents: 48660
diff changeset
468 else
50540
4229a4b14f1d Merged revisions 67653,67655,67682-67683,67724,67755,67780-67783,67786,67789,67841,67843,67865,67881,67884,67925,67929,67931 via svnmerge from
Georg Brandl <georg@python.org>
parents: 49539
diff changeset
469 stream = PyObject_CallMethod(io, "open", "isisOOO",
4229a4b14f1d Merged revisions 67653,67655,67682-67683,67724,67755,67780-67783,67786,67789,67841,67843,67865,67881,67884,67925,67929,67931 via svnmerge from
Georg Brandl <georg@python.org>
parents: 49539
diff changeset
470 fileno(tok->fp), "r", -1, enc, Py_None, Py_None, Py_False);
41970
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
471 if (stream == NULL)
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
472 goto cleanup;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
473
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
474 Py_XDECREF(tok->decoding_readline);
41970
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
475 readline = PyObject_GetAttrString(stream, "readline");
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
476 tok->decoding_readline = readline;
41970
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
477
49484
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
478 /* The file has been reopened; parsing will restart from
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
479 * the beginning of the file, we have to reset the line number.
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
480 * But this function has been called from inside tok_nextc() which
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
481 * will increment lineno before it returns. So we set it -1 so that
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
482 * the next call to tok_nextc() will start with tok->lineno == 0.
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
483 */
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
484 tok->lineno = -1;
0b7e023803fb Issues #2384 and #3975: Tracebacks were not correctly printed when the source file
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49021
diff changeset
485
41970
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
486 cleanup:
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
487 Py_XDECREF(stream);
b3900a51346b tokenizer.c: make coding markup work again.
Guido van Rossum <guido@python.org>
parents: 41670
diff changeset
488 Py_XDECREF(io);
42551
b6625d2b378a Fix refleaks from execfile('file that contains a # coding: line')
Neal Norwitz <nnorwitz@gmail.com>
parents: 42548
diff changeset
489 return readline != NULL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
490 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
491
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
492 /* Fetch the next byte from TOK. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
493
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
494 static int fp_getc(struct tok_state *tok) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
495 return getc(tok->fp);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
496 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
497
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
498 /* Unfetch the last byte back into TOK. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
499
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
500 static void fp_ungetc(int c, struct tok_state *tok) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
501 ungetc(c, tok->fp);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
502 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
503
42403
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
504 /* Check whether the characters at s start a valid
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
505 UTF-8 sequence. Return the number of characters forming
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
506 the sequence if yes, 0 if not. */
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
507 static int valid_utf8(const unsigned char* s)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
508 {
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
509 int expected = 0;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
510 int length;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
511 if (*s < 0x80)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
512 /* single-byte code */
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
513 return 1;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
514 if (*s < 0xc0)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
515 /* following byte */
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
516 return 0;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
517 if (*s < 0xE0)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
518 expected = 1;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
519 else if (*s < 0xF0)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
520 expected = 2;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
521 else if (*s < 0xF8)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
522 expected = 3;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
523 else
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
524 return 0;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
525 length = expected + 1;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
526 for (; expected; expected--)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
527 if (s[expected] < 0x80 || s[expected] >= 0xC0)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
528 return 0;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
529 return length;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
530 }
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
531
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
532 /* Read a line of input from TOK. Determine encoding
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
533 if necessary. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
534
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
535 static char *
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
536 decoding_fgets(char *s, int size, struct tok_state *tok)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
537 {
24630
43caa212b959 Use Py_FatalError instead of abort.
Martin v. Löwis <martin@v.loewis.de>
parents: 24624
diff changeset
538 char *line = NULL;
36741
c0982b8155c6 Change non-ASCII warning into a SyntaxError.
Martin v. Löwis <martin@v.loewis.de>
parents: 36520
diff changeset
539 int badchar = 0;
24543
c00df23e856e Group statements properly.
Martin v. Löwis <martin@v.loewis.de>
parents: 24542
diff changeset
540 for (;;) {
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
541 if (tok->decoding_state == STATE_NORMAL) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
542 /* We already have a codec associated with
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
543 this input. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
544 line = fp_readl(s, size, tok);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
545 break;
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
546 } else if (tok->decoding_state == STATE_RAW) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
547 /* We want a 'raw' read. */
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
548 line = Py_UniversalNewlineFgets(s, size,
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
549 tok->fp, NULL);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
550 break;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
551 } else {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
552 /* We have not yet determined the encoding.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
553 If an encoding is found, use the file-pointer
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
554 reader functions from now on. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
555 if (!check_bom(fp_getc, fp_ungetc, fp_setreadl, tok))
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
556 return error_ret(tok);
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
557 assert(tok->decoding_state != STATE_INIT);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
558 }
24543
c00df23e856e Group statements properly.
Martin v. Löwis <martin@v.loewis.de>
parents: 24542
diff changeset
559 }
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
560 if (line != NULL && tok->lineno < 2 && !tok->read_coding_spec) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
561 if (!check_coding_spec(line, strlen(line), tok, fp_setreadl)) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
562 return error_ret(tok);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
563 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
564 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
565 #ifndef PGEN
42403
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
566 /* The default encoding is UTF-8, so make sure we don't have any
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
567 non-UTF-8 sequences in it. */
36741
c0982b8155c6 Change non-ASCII warning into a SyntaxError.
Martin v. Löwis <martin@v.loewis.de>
parents: 36520
diff changeset
568 if (line && !tok->encoding) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
569 unsigned char *c;
42403
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
570 int length;
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
571 for (c = (unsigned char *)line; *c; c += length)
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
572 if (!(length = valid_utf8(c))) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
573 badchar = *c;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
574 break;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
575 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
576 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
577 if (badchar) {
27918
be371ffc4cd0 Add URL for PEP to the source code encoding warning.
Marc-André Lemburg <mal@egenix.com>
parents: 27772
diff changeset
578 char buf[500];
24558
d26b62f0c6db Add 1 to lineno in deprecation warning. Fixes #590888.
Martin v. Löwis <martin@v.loewis.de>
parents: 24544
diff changeset
579 /* Need to add 1 to the line number, since this line
d26b62f0c6db Add 1 to lineno in deprecation warning. Fixes #590888.
Martin v. Löwis <martin@v.loewis.de>
parents: 24544
diff changeset
580 has not been counted, yet. */
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
581 sprintf(buf,
42403
7e8469617b1c Implement PEP 3120.
Martin v. Löwis <martin@v.loewis.de>
parents: 42084
diff changeset
582 "Non-UTF-8 code starting with '\\x%.2x' "
27918
be371ffc4cd0 Add URL for PEP to the source code encoding warning.
Marc-André Lemburg <mal@egenix.com>
parents: 27772
diff changeset
583 "in file %.200s on line %i, "
be371ffc4cd0 Add URL for PEP to the source code encoding warning.
Marc-André Lemburg <mal@egenix.com>
parents: 27772
diff changeset
584 "but no encoding declared; "
43018
165934bc4429 Fix an outdated URL in a SyntaxError message.
Guido van Rossum <guido@python.org>
parents: 43002
diff changeset
585 "see http://python.org/dev/peps/pep-0263/ for details",
27918
be371ffc4cd0 Add URL for PEP to the source code encoding warning.
Marc-André Lemburg <mal@egenix.com>
parents: 27772
diff changeset
586 badchar, tok->filename, tok->lineno + 1);
36741
c0982b8155c6 Change non-ASCII warning into a SyntaxError.
Martin v. Löwis <martin@v.loewis.de>
parents: 36520
diff changeset
587 PyErr_SetString(PyExc_SyntaxError, buf);
c0982b8155c6 Change non-ASCII warning into a SyntaxError.
Martin v. Löwis <martin@v.loewis.de>
parents: 36520
diff changeset
588 return error_ret(tok);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
589 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
590 #endif
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
591 return line;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
592 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
593
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
594 static int
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
595 decoding_feof(struct tok_state *tok)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
596 {
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
597 if (tok->decoding_state != STATE_NORMAL) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
598 return feof(tok->fp);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
599 } else {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
600 PyObject* buf = tok->decoding_buffer;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
601 if (buf == NULL) {
35231
763e85b107b4 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
Walter Dörwald <walter@livinglogic.de>
parents: 34664
diff changeset
602 buf = PyObject_CallObject(tok->decoding_readline, NULL);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
603 if (buf == NULL) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
604 error_ret(tok);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
605 return 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
606 } else {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
607 tok->decoding_buffer = buf;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
608 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
609 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
610 return PyObject_Length(buf) == 0;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
611 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
612 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
613
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
614 /* Fetch a byte from TOK, using the string buffer. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
615
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
616 static int
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
617 buf_getc(struct tok_state *tok) {
27772
3566b84e66b9 patch 680474 that fixes bug 679880: compile/eval/exec refused utf-8 bom
Just van Rossum <just@letterror.com>
parents: 27247
diff changeset
618 return Py_CHARMASK(*tok->str++);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
619 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
620
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
621 /* Unfetch a byte from TOK, using the string buffer. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
622
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
623 static void
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
624 buf_ungetc(int c, struct tok_state *tok) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
625 tok->str--;
27772
3566b84e66b9 patch 680474 that fixes bug 679880: compile/eval/exec refused utf-8 bom
Just van Rossum <just@letterror.com>
parents: 27247
diff changeset
626 assert(Py_CHARMASK(*tok->str) == c); /* tok->cur may point to read-only segment */
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
627 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
628
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
629 /* Set the readline function for TOK to ENC. For the string-based
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
630 tokenizer, this means to just record the encoding. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
631
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
632 static int
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
633 buf_setreadl(struct tok_state *tok, const char* enc) {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
634 tok->enc = enc;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
635 return 1;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
636 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
637
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
638 /* Return a UTF-8 encoding Python string object from the
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
639 C byte string STR, which is encoded with ENC. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
640
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
641 static PyObject *
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
642 translate_into_utf8(const char* str, const char* enc) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
643 PyObject *utf8;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
644 PyObject* buf = PyUnicode_Decode(str, strlen(str), enc, NULL);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
645 if (buf == NULL)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
646 return NULL;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
647 utf8 = PyUnicode_AsUTF8String(buf);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
648 Py_DECREF(buf);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
649 return utf8;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
650 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
651
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
652
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
653 static char *
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
654 translate_newlines(const char *s, int exec_input, struct tok_state *tok) {
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
655 int skip_next_lf = 0, needed_length = strlen(s) + 2, final_length;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
656 char *buf, *current;
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
657 char c = '\0';
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
658 buf = PyMem_MALLOC(needed_length);
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
659 if (buf == NULL) {
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
660 tok->done = E_NOMEM;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
661 return NULL;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
662 }
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
663 for (current = buf; *s; s++, current++) {
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
664 c = *s;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
665 if (skip_next_lf) {
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
666 skip_next_lf = 0;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
667 if (c == '\n') {
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
668 c = *++s;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
669 if (!c)
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
670 break;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
671 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
672 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
673 if (c == '\r') {
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
674 skip_next_lf = 1;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
675 c = '\n';
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
676 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
677 *current = c;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
678 }
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
679 /* If this is exec input, add a newline to the end of the string if
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
680 there isn't one already. */
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
681 if (exec_input && c != '\n') {
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
682 *current = '\n';
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
683 current++;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
684 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
685 *current = '\0';
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
686 final_length = current - buf + 1;
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
687 if (final_length < needed_length && final_length)
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
688 /* should never fail */
57678
a769a9b34fbc Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57209
diff changeset
689 buf = PyMem_REALLOC(buf, final_length);
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
690 return buf;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
691 }
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
692
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
693 /* Decode a byte string STR for use as the buffer of TOK.
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
694 Look for encoding declarations inside STR, and record them
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
695 inside TOK. */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
696
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
697 static const char *
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
698 decode_str(const char *input, int single, struct tok_state *tok)
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
699 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
700 PyObject* utf8 = NULL;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
701 const char *str;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
702 const char *s;
44885
fefaa0e66fdd Merged revisions 60176-60209 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 44859
diff changeset
703 const char *newl[2] = {NULL, NULL};
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
704 int lineno = 0;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
705 tok->input = str = translate_newlines(input, single, tok);
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
706 if (str == NULL)
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
707 return NULL;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
708 tok->enc = NULL;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
709 tok->str = str;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
710 if (!check_bom(buf_getc, buf_ungetc, buf_setreadl, tok))
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
711 return error_ret(tok);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
712 str = tok->str; /* string after BOM if any */
24542
d0d3f760d067 Repaired a fatal compiler error in the debug build: it's not clear what
Tim Peters <tim.peters@gmail.com>
parents: 24541
diff changeset
713 assert(str);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
714 if (tok->enc != NULL) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
715 utf8 = translate_into_utf8(str, tok->enc);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
716 if (utf8 == NULL)
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
717 return error_ret(tok);
47301
23ab586c427a Renamed PyString to PyBytes
Christian Heimes <christian@cheimes.de>
parents: 47300
diff changeset
718 str = PyBytes_AsString(utf8);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
719 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
720 for (s = str;; s++) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
721 if (*s == '\0') break;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
722 else if (*s == '\n') {
45017
4725c84b0422 Merged revisions 60350-60363 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 44885
diff changeset
723 assert(lineno < 2);
44859
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
724 newl[lineno] = s;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
725 lineno++;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
726 if (lineno == 2) break;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
727 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
728 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
729 tok->enc = NULL;
44859
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
730 /* need to check line 1 and 2 separately since check_coding_spec
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
731 assumes a single line as input */
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
732 if (newl[0]) {
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
733 if (!check_coding_spec(str, newl[0] - str, tok, buf_setreadl))
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
734 return error_ret(tok);
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
735 if (tok->enc == NULL && newl[1]) {
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
736 if (!check_coding_spec(newl[0]+1, newl[1] - newl[0],
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
737 tok, buf_setreadl))
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
738 return error_ret(tok);
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
739 }
feba5d5cac22 Merged revisions 60151-60159,60161-60168,60170,60172-60173,60175 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44782
diff changeset
740 }
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
741 if (tok->enc != NULL) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
742 assert(utf8 == NULL);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
743 utf8 = translate_into_utf8(str, tok->enc);
54917
f94c09aa4444 Merged revisions 73623-73624 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 52641
diff changeset
744 if (utf8 == NULL)
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
745 return error_ret(tok);
47301
23ab586c427a Renamed PyString to PyBytes
Christian Heimes <christian@cheimes.de>
parents: 47300
diff changeset
746 str = PyBytes_AS_STRING(utf8);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
747 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
748 assert(tok->decoding_buffer == NULL);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
749 tok->decoding_buffer = utf8; /* CAUTION */
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
750 return str;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
751 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
752
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
753 #endif /* PGEN */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
754
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
755 /* Set up tokenizer for string */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
756
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
757 struct tok_state *
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
758 PyTokenizer_FromString(const char *str, int exec_input)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
759 {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
760 struct tok_state *tok = tok_new();
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
761 if (tok == NULL)
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
762 return NULL;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
763 str = (char *)decode_str(str, exec_input, tok);
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
764 if (str == NULL) {
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
765 PyTokenizer_Free(tok);
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
766 return NULL;
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
767 }
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
768
26601
cc05c2c8660e Constify filenames and scripts. Fixes #651362.
Martin v. Löwis <martin@v.loewis.de>
parents: 25995
diff changeset
769 /* XXX: constify members. */
cc05c2c8660e Constify filenames and scripts. Fixes #651362.
Martin v. Löwis <martin@v.loewis.de>
parents: 25995
diff changeset
770 tok->buf = tok->cur = tok->end = tok->inp = (char*)str;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
771 return tok;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
772 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
773
51956
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
774 struct tok_state *
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
775 PyTokenizer_FromUTF8(const char *str, int exec_input)
51956
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
776 {
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
777 struct tok_state *tok = tok_new();
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
778 if (tok == NULL)
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
779 return NULL;
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
780 #ifndef PGEN
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
781 tok->input = str = translate_newlines(str, exec_input, tok);
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
782 #endif
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
783 if (str == NULL) {
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
784 PyTokenizer_Free(tok);
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
785 return NULL;
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
786 }
51956
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
787 tok->decoding_state = STATE_RAW;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
788 tok->read_coding_spec = 1;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
789 tok->enc = NULL;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
790 tok->str = str;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
791 tok->encoding = (char *)PyMem_MALLOC(6);
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
792 if (!tok->encoding) {
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
793 PyTokenizer_Free(tok);
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
794 return NULL;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
795 }
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
796 strcpy(tok->encoding, "utf-8");
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
797
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
798 /* XXX: constify members. */
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
799 tok->buf = tok->cur = tok->end = tok->inp = (char*)str;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
800 return tok;
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
801 }
8fe2988cf550 ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626
Benjamin Peterson <benjamin@python.org>
parents: 50540
diff changeset
802
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
803 /* Set up tokenizer for file */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
804
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
805 struct tok_state *
43225
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
806 PyTokenizer_FromFile(FILE *fp, char* enc, char *ps1, char *ps2)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
807 {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
808 struct tok_state *tok = tok_new();
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
809 if (tok == NULL)
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
810 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
811 if ((tok->buf = (char *)PyMem_MALLOC(BUFSIZ)) == NULL) {
35878
94a409da4bd4 Fix some more memory leaks.
Neal Norwitz <nnorwitz@gmail.com>
parents: 35751
diff changeset
812 PyTokenizer_Free(tok);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
813 return NULL;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
814 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
815 tok->cur = tok->inp = tok->buf;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
816 tok->end = tok->buf + BUFSIZ;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
817 tok->fp = fp;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
818 tok->prompt = ps1;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
819 tok->nextprompt = ps2;
43225
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
820 if (enc != NULL) {
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
821 /* Must copy encoding declaration since it
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
822 gets copied into the parse tree. */
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
823 tok->encoding = PyMem_MALLOC(strlen(enc)+1);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
824 if (!tok->encoding) {
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
825 PyTokenizer_Free(tok);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
826 return NULL;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
827 }
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
828 strcpy(tok->encoding, enc);
43421
f31d415554cd Use an enum for decoding_state. It makes the code a little more
Neil Schemenauer <nascheme@enme.ucalgary.ca>
parents: 43292
diff changeset
829 tok->decoding_state = STATE_NORMAL;
43225
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
830 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
831 return tok;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
832 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
833
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
834
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
835 /* Free a tok_state structure */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
836
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
837 void
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
838 PyTokenizer_Free(struct tok_state *tok)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
839 {
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
840 if (tok->encoding != NULL)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
841 PyMem_FREE(tok->encoding);
24544
0bd521eb714e Make pgen compile with pydebug. Duplicate normalized names, as it may
Martin v. Löwis <martin@v.loewis.de>
parents: 24543
diff changeset
842 #ifndef PGEN
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
843 Py_XDECREF(tok->decoding_readline);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
844 Py_XDECREF(tok->decoding_buffer);
24544
0bd521eb714e Make pgen compile with pydebug. Duplicate normalized names, as it may
Martin v. Löwis <martin@v.loewis.de>
parents: 24543
diff changeset
845 #endif
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
846 if (tok->fp != NULL && tok->buf != NULL)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
847 PyMem_FREE(tok->buf);
57206
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
848 if (tok->input)
b2d90c1c637d Merged revisions 76230 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 56594
diff changeset
849 PyMem_FREE((char *)tok->input);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
850 PyMem_FREE(tok);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
851 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
852
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
853 /* Get next char, updating state; error code goes into tok->done */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
854
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
855 static int
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
856 tok_nextc(register struct tok_state *tok)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
857 {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
858 for (;;) {
1897
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
859 if (tok->cur != tok->inp) {
7118
e9108e97a5a0 tok_nextc() should return unsigned characters, to avoid mistaking
Guido van Rossum <guido@python.org>
parents: 5379
diff changeset
860 return Py_CHARMASK(*tok->cur++); /* Fast path */
1897
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
861 }
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
862 if (tok->done != E_OK)
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
863 return EOF;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
864 if (tok->fp == NULL) {
1897
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
865 char *end = strchr(tok->inp, '\n');
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
866 if (end != NULL)
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
867 end++;
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
868 else {
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
869 end = strchr(tok->inp, '\0');
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
870 if (end == tok->inp) {
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
871 tok->done = E_EOF;
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
872 return EOF;
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
873 }
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
874 }
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
875 if (tok->start == NULL)
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
876 tok->buf = tok->cur;
36786
57322cb2ca11 Patch #1440601: Add col_offset attribute to AST nodes.
Martin v. Löwis <martin@v.loewis.de>
parents: 36741
diff changeset
877 tok->line_start = tok->cur;
1897
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
878 tok->lineno++;
3fbda8768855 * Parser/tokenizer.c (tok_nextc): count line numbers when parsing
Guido van Rossum <guido@python.org>
parents: 1835
diff changeset
879 tok->inp = end;
7118
e9108e97a5a0 tok_nextc() should return unsigned characters, to avoid mistaking
Guido van Rossum <guido@python.org>
parents: 5379
diff changeset
880 return Py_CHARMASK(*tok->cur++);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
881 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
882 if (tok->prompt != NULL) {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
883 char *newtok = PyOS_Readline(stdin, stdout, tok->prompt);
43225
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
884 #ifndef PGEN
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
885 if (tok->encoding && newtok && *newtok) {
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
886 /* Recode to UTF-8 */
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
887 Py_ssize_t buflen;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
888 const char* buf;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
889 PyObject *u = translate_into_utf8(newtok, tok->encoding);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
890 PyMem_FREE(newtok);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
891 if (!u) {
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
892 tok->done = E_DECODE;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
893 return EOF;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
894 }
47301
23ab586c427a Renamed PyString to PyBytes
Christian Heimes <christian@cheimes.de>
parents: 47300
diff changeset
895 buflen = PyBytes_GET_SIZE(u);
23ab586c427a Renamed PyString to PyBytes
Christian Heimes <christian@cheimes.de>
parents: 47300
diff changeset
896 buf = PyBytes_AS_STRING(u);
43225
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
897 if (!buf) {
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
898 Py_DECREF(u);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
899 tok->done = E_DECODE;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
900 return EOF;
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
901 }
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
902 newtok = PyMem_MALLOC(buflen+1);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
903 strcpy(newtok, buf);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
904 Py_DECREF(u);
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
905 }
38f3f9ef193d Convert code from sys.stdin.encoding to UTF-8 in
Martin v. Löwis <martin@v.loewis.de>
parents: 43018
diff changeset
906 #endif
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
907 if (tok->nextprompt != NULL)
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
908 tok->prompt = tok->nextprompt;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
909 if (newtok == NULL)
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
910 tok->done = E_INTR;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
911 else if (*newtok == '\0') {
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
912 PyMem_FREE(newtok);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
913 tok->done = E_EOF;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
914 }
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
915 else if (tok->start != NULL) {
13093
cbe02b8aee07 Trent Mick: familiar simple Win64 patches
Guido van Rossum <guido@python.org>
parents: 12576
diff changeset
916 size_t start = tok->start - tok->buf;
cbe02b8aee07 Trent Mick: familiar simple Win64 patches
Guido van Rossum <guido@python.org>
parents: 12576
diff changeset
917 size_t oldlen = tok->cur - tok->buf;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
918 size_t newlen = oldlen + strlen(newtok);
12576
1c7c2dd1beb1 Vladimir Marangozov's long-awaited malloc restructuring.
Guido van Rossum <guido@python.org>
parents: 12224
diff changeset
919 char *buf = tok->buf;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
920 buf = (char *)PyMem_REALLOC(buf, newlen+1);
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
921 tok->lineno++;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
922 if (buf == NULL) {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
923 PyMem_FREE(tok->buf);
2044
281e40652e23 Parser/tokenizer.c (tok_nextc): zap tok->buf after freeing;
Guido van Rossum <guido@python.org>
parents: 1897
diff changeset
924 tok->buf = NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
925 PyMem_FREE(newtok);
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
926 tok->done = E_NOMEM;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
927 return EOF;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
928 }
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
929 tok->buf = buf;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
930 tok->cur = tok->buf + oldlen;
36786
57322cb2ca11 Patch #1440601: Add col_offset attribute to AST nodes.
Martin v. Löwis <martin@v.loewis.de>
parents: 36741
diff changeset
931 tok->line_start = tok->cur;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
932 strcpy(tok->buf + oldlen, newtok);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
933 PyMem_FREE(newtok);
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
934 tok->inp = tok->buf + newlen;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
935 tok->end = tok->inp + 1;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
936 tok->start = tok->buf + start;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
937 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
938 else {
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
939 tok->lineno++;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
940 if (tok->buf != NULL)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
941 PyMem_FREE(tok->buf);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
942 tok->buf = newtok;
36786
57322cb2ca11 Patch #1440601: Add col_offset attribute to AST nodes.
Martin v. Löwis <martin@v.loewis.de>
parents: 36741
diff changeset
943 tok->line_start = tok->buf;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
944 tok->cur = tok->buf;
36786
57322cb2ca11 Patch #1440601: Add col_offset attribute to AST nodes.
Martin v. Löwis <martin@v.loewis.de>
parents: 36741
diff changeset
945 tok->line_start = tok->buf;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
946 tok->inp = strchr(tok->buf, '\0');
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
947 tok->end = tok->inp + 1;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
948 }
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
949 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
950 else {
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
951 int done = 0;
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
952 Py_ssize_t cur = 0;
2732
daa150b3a6e4 replace "\r\n" with "\n" at line end (Jim Ahlstrom)
Guido van Rossum <guido@python.org>
parents: 2149
diff changeset
953 char *pt;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
954 if (tok->start == NULL) {
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
955 if (tok->buf == NULL) {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
956 tok->buf = (char *)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
957 PyMem_MALLOC(BUFSIZ);
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
958 if (tok->buf == NULL) {
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
959 tok->done = E_NOMEM;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
960 return EOF;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
961 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
962 tok->end = tok->buf + BUFSIZ;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
963 }
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
964 if (decoding_fgets(tok->buf, (int)(tok->end - tok->buf),
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
965 tok) == NULL) {
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
966 tok->done = E_EOF;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
967 done = 1;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
968 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
969 else {
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
970 tok->done = E_OK;
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
971 tok->inp = strchr(tok->buf, '\0');
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
972 done = tok->inp[-1] == '\n';
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
973 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
974 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
975 else {
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
976 cur = tok->cur - tok->buf;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
977 if (decoding_feof(tok)) {
2149
6060d7fb07e3 fix loop on unterminated triple quotes
Guido van Rossum <guido@python.org>
parents: 2060
diff changeset
978 tok->done = E_EOF;
6060d7fb07e3 fix loop on unterminated triple quotes
Guido van Rossum <guido@python.org>
parents: 2060
diff changeset
979 done = 1;
6060d7fb07e3 fix loop on unterminated triple quotes
Guido van Rossum <guido@python.org>
parents: 2060
diff changeset
980 }
6060d7fb07e3 fix loop on unterminated triple quotes
Guido van Rossum <guido@python.org>
parents: 2060
diff changeset
981 else
6060d7fb07e3 fix loop on unterminated triple quotes
Guido van Rossum <guido@python.org>
parents: 2060
diff changeset
982 tok->done = E_OK;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
983 }
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
984 tok->lineno++;
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
985 /* Read until '\n' or EOF */
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
986 while (!done) {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
987 Py_ssize_t curstart = tok->start == NULL ? -1 :
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
988 tok->start - tok->buf;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
989 Py_ssize_t curvalid = tok->inp - tok->buf;
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 36090
diff changeset
990 Py_ssize_t newsize = curvalid + BUFSIZ;
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
991 char *newbuf = tok->buf;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
992 newbuf = (char *)PyMem_REALLOC(newbuf,
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
993 newsize);
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
994 if (newbuf == NULL) {
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
995 tok->done = E_NOMEM;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
996 tok->cur = tok->inp;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
997 return EOF;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
998 }
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
999 tok->buf = newbuf;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1000 tok->inp = tok->buf + curvalid;
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1001 tok->end = tok->buf + newsize;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1002 tok->start = curstart < 0 ? NULL :
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1003 tok->buf + curstart;
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1004 if (decoding_fgets(tok->inp,
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1005 (int)(tok->end - tok->inp),
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1006 tok) == NULL) {
36809
2c2dffe070df Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes
Thomas Wouters <thomas@python.org>
parents: 36786
diff changeset
1007 /* Break out early on decoding
2c2dffe070df Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes
Thomas Wouters <thomas@python.org>
parents: 36786
diff changeset
1008 errors, as tok->buf will be NULL
2c2dffe070df Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes
Thomas Wouters <thomas@python.org>
parents: 36786
diff changeset
1009 */
2c2dffe070df Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes
Thomas Wouters <thomas@python.org>
parents: 36786
diff changeset
1010 if (tok->decoding_erred)
2c2dffe070df Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes
Thomas Wouters <thomas@python.org>
parents: 36786
diff changeset
1011 return EOF;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1012 /* Last line does not end in \n,
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1013 fake one */
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1014 strcpy(tok->inp, "\n");
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1015 }
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1016 tok->inp = strchr(tok->inp, '\0');
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1017 done = tok->inp[-1] == '\n';
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1018 }
38669
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1019 if (tok->buf != NULL) {
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1020 tok->cur = tok->buf + cur;
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1021 tok->line_start = tok->cur;
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1022 /* replace "\r\n" with "\n" */
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 39676
diff changeset
1023 /* For Mac leave the \r, giving a syntax error */
38669
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1024 pt = tok->inp - 2;
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1025 if (pt >= tok->buf && *pt == '\r') {
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1026 *pt++ = '\n';
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1027 *pt = '\0';
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1028 tok->inp = pt;
19fd6141940d Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
1029 }
2732
daa150b3a6e4 replace "\r\n" with "\n" at line end (Jim Ahlstrom)
Guido van Rossum <guido@python.org>
parents: 2149
diff changeset
1030 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1031 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1032 if (tok->done != E_OK) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1033 if (tok->prompt != NULL)
9047
6d2eb58f0418 Replace all calls to fprintf(stderr, ...) with PySys_WriteStderr(...).
Guido van Rossum <guido@python.org>
parents: 7945
diff changeset
1034 PySys_WriteStderr("\n");
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1035 tok->cur = tok->inp;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1036 return EOF;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1037 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1038 }
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1039 /*NOTREACHED*/
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1040 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1041
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1042
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1043 /* Back-up one character */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1044
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1045 static void
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
1046 tok_backup(register struct tok_state *tok, register int c)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1047 {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1048 if (c != EOF) {
2044
281e40652e23 Parser/tokenizer.c (tok_nextc): zap tok->buf after freeing;
Guido van Rossum <guido@python.org>
parents: 1897
diff changeset
1049 if (--tok->cur < tok->buf)
57209
b354c1b6b476 Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 57206
diff changeset
1050 Py_FatalError("tok_backup: beginning of buffer");
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1051 if (*tok->cur != c)
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1052 *tok->cur = c;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1053 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1054 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1055
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1056
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1057 /* Return the token corresponding to a single character */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1058
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1059 int
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
1060 PyToken_OneChar(int c)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1061 {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1062 switch (c) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1063 case '(': return LPAR;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1064 case ')': return RPAR;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1065 case '[': return LSQB;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1066 case ']': return RSQB;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1067 case ':': return COLON;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1068 case ',': return COMMA;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1069 case ';': return SEMI;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1070 case '+': return PLUS;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1071 case '-': return MINUS;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1072 case '*': return STAR;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1073 case '/': return SLASH;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1074 case '|': return VBAR;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1075 case '&': return AMPER;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1076 case '<': return LESS;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1077 case '>': return GREATER;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1078 case '=': return EQUAL;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1079 case '.': return DOT;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1080 case '%': return PERCENT;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1081 case '{': return LBRACE;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1082 case '}': return RBRACE;
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1083 case '^': return CIRCUMFLEX;
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1084 case '~': return TILDE;
32888
c7e03cf8333e PEP-0318, @decorator-style. In Guido's words:
Anthony Baxter <anthonybaxter@gmail.com>
parents: 30979
diff changeset
1085 case '@': return AT;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1086 default: return OP;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1087 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1088 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1089
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1090
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1091 int
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
1092 PyToken_TwoChars(int c1, int c2)
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1093 {
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1094 switch (c1) {
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1095 case '=':
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1096 switch (c2) {
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1097 case '=': return EQEQUAL;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1098 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1099 break;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1100 case '!':
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1101 switch (c2) {
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1102 case '=': return NOTEQUAL;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1103 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1104 break;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1105 case '<':
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1106 switch (c2) {
52641
0530aadff696 The BDFL has retired! Long live the FLUFL (Friendly Language Uncle For Life)!
Brett Cannon <bcannon@gmail.com>
parents: 51956
diff changeset
1107 case '>': return NOTEQUAL;
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1108 case '=': return LESSEQUAL;
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1109 case '<': return LEFTSHIFT;
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1110 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1111 break;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1112 case '>':
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1113 switch (c2) {
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1114 case '=': return GREATEREQUAL;
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1115 case '>': return RIGHTSHIFT;
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1116 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1117 break;
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1118 case '+':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1119 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1120 case '=': return PLUSEQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1121 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1122 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1123 case '-':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1124 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1125 case '=': return MINEQUAL;
40529
54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds
Neal Norwitz <nnorwitz@gmail.com>
parents: 40450
diff changeset
1126 case '>': return RARROW;
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1127 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1128 break;
3333
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1129 case '*':
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1130 switch (c2) {
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1131 case '*': return DOUBLESTAR;
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1132 case '=': return STAREQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1133 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1134 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1135 case '/':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1136 switch (c2) {
19051
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1137 case '/': return DOUBLESLASH;
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1138 case '=': return SLASHEQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1139 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1140 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1141 case '|':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1142 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1143 case '=': return VBAREQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1144 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1145 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1146 case '%':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1147 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1148 case '=': return PERCENTEQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1149 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1150 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1151 case '&':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1152 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1153 case '=': return AMPEREQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1154 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1155 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1156 case '^':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1157 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1158 case '=': return CIRCUMFLEXEQUAL;
3333
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1159 }
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1160 break;
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1161 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1162 return OP;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1163 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1164
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1165 int
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1166 PyToken_ThreeChars(int c1, int c2, int c3)
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1167 {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1168 switch (c1) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1169 case '<':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1170 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1171 case '<':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1172 switch (c3) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1173 case '=':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1174 return LEFTSHIFTEQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1175 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1176 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1177 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1178 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1179 case '>':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1180 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1181 case '>':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1182 switch (c3) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1183 case '=':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1184 return RIGHTSHIFTEQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1185 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1186 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1187 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1188 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1189 case '*':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1190 switch (c2) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1191 case '*':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1192 switch (c3) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1193 case '=':
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1194 return DOUBLESTAREQUAL;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1195 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1196 break;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1197 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1198 break;
19051
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1199 case '/':
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1200 switch (c2) {
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1201 case '/':
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1202 switch (c3) {
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1203 case '=':
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1204 return DOUBLESLASHEQUAL;
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1205 }
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1206 break;
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1207 }
f20727cc3d0f Implement PEP 238 in its (almost) full glory.
Guido van Rossum <guido@python.org>
parents: 17600
diff changeset
1208 break;
41230
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1209 case '.':
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1210 switch (c2) {
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1211 case '.':
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1212 switch (c3) {
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1213 case '.':
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1214 return ELLIPSIS;
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1215 }
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1216 break;
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1217 }
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1218 break;
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1219 }
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1220 return OP;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1221 }
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1222
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1223 static int
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
1224 indenterror(struct tok_state *tok)
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1225 {
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1226 if (tok->alterror) {
13539
feab677269e0 Create two new exceptions: IndentationError and TabError. These are
Fred Drake <fdrake@acm.org>
parents: 13447
diff changeset
1227 tok->done = E_TABSPACE;
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1228 tok->cur = tok->inp;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1229 return 1;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1230 }
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1231 if (tok->altwarning) {
13539
feab677269e0 Create two new exceptions: IndentationError and TabError. These are
Fred Drake <fdrake@acm.org>
parents: 13447
diff changeset
1232 PySys_WriteStderr("%s: inconsistent use of tabs and spaces "
feab677269e0 Create two new exceptions: IndentationError and TabError. These are
Fred Drake <fdrake@acm.org>
parents: 13447
diff changeset
1233 "in indentation\n", tok->filename);
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1234 tok->altwarning = 0;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1235 }
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1236 return 0;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1237 }
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1238
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1239 #ifdef PGEN
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1240 #define verify_identifier(s,e) 1
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1241 #else
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1242 /* Verify that the identifier follows PEP 3131. */
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1243 static int
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1244 verify_identifier(char *start, char *end)
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1245 {
43002
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1246 PyObject *s;
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1247 int result;
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1248 s = PyUnicode_DecodeUTF8(start, end-start, NULL);
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1249 if (s == NULL) {
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1250 PyErr_Clear();
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1251 return 0;
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1252 }
e142cd4b5710 Fix issue # 1037 (sort of).
Guido van Rossum <guido@python.org>
parents: 42984
diff changeset
1253 result = PyUnicode_IsIdentifier(s);
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1254 Py_DECREF(s);
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1255 return result;
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1256 }
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1257 #endif
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1258
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1259 /* Get next token, after space stripping etc. */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1260
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1261 static int
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1262 tok_get(register struct tok_state *tok, char **p_start, char **p_end)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1263 {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1264 register int c;
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1265 int blankline, nonascii;
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1266
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1267 *p_start = *p_end = NULL;
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1268 nextline:
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1269 tok->start = NULL;
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1270 blankline = 0;
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1271
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1272 /* Get indentation level */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1273 if (tok->atbol) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1274 register int col = 0;
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1275 register int altcol = 0;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1276 tok->atbol = 0;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1277 for (;;) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1278 c = tok_nextc(tok);
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1279 if (c == ' ')
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1280 col++, altcol++;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1281 else if (c == '\t') {
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1282 col = (col/tok->tabsize + 1) * tok->tabsize;
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1283 altcol = (altcol/tok->alttabsize + 1)
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1284 * tok->alttabsize;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1285 }
2801
e911f2fb179c ignore control-l in whitespace
Guido van Rossum <guido@python.org>
parents: 2732
diff changeset
1286 else if (c == '\014') /* Control-L (formfeed) */
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1287 col = altcol = 0; /* For Emacs users */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1288 else
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1289 break;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1290 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1291 tok_backup(tok, c);
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1292 if (c == '#' || c == '\n') {
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1293 /* Lines with only whitespace and/or comments
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1294 shouldn't affect the indentation and are
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1295 not passed to the parser as NEWLINE tokens,
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1296 except *totally* empty lines in interactive
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1297 mode, which signal the end of a command group. */
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1298 if (col == 0 && c == '\n' && tok->prompt != NULL)
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1299 blankline = 0; /* Let it through */
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1300 else
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1301 blankline = 1; /* Ignore completely */
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1302 /* We can't jump back right here since we still
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1303 may need to skip to the end of a comment */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1304 }
1284
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1305 if (!blankline && tok->level == 0) {
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1306 if (col == tok->indstack[tok->indent]) {
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1307 /* No change */
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1308 if (altcol != tok->altindstack[tok->indent]) {
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1309 if (indenterror(tok))
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1310 return ERRORTOKEN;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1311 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1312 }
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1313 else if (col > tok->indstack[tok->indent]) {
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1314 /* Indent -- always one */
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1315 if (tok->indent+1 >= MAXINDENT) {
13539
feab677269e0 Create two new exceptions: IndentationError and TabError. These are
Fred Drake <fdrake@acm.org>
parents: 13447
diff changeset
1316 tok->done = E_TOODEEP;
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1317 tok->cur = tok->inp;
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1318 return ERRORTOKEN;
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1319 }
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1320 if (altcol <= tok->altindstack[tok->indent]) {
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1321 if (indenterror(tok))
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1322 return ERRORTOKEN;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1323 }
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1324 tok->pendin++;
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1325 tok->indstack[++tok->indent] = col;
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1326 tok->altindstack[tok->indent] = altcol;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1327 }
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1328 else /* col < tok->indstack[tok->indent] */ {
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1329 /* Dedent -- any number, must be consistent */
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1330 while (tok->indent > 0 &&
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1331 col < tok->indstack[tok->indent]) {
7324
55125b1144fa Swap two statements in the dedent check loop. This makes absolutely
Guido van Rossum <guido@python.org>
parents: 7315
diff changeset
1332 tok->pendin--;
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1333 tok->indent--;
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1334 }
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1335 if (col != tok->indstack[tok->indent]) {
13539
feab677269e0 Create two new exceptions: IndentationError and TabError. These are
Fred Drake <fdrake@acm.org>
parents: 13447
diff changeset
1336 tok->done = E_DEDENT;
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1337 tok->cur = tok->inp;
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1338 return ERRORTOKEN;
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1339 }
7945
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1340 if (altcol != tok->altindstack[tok->indent]) {
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1341 if (indenterror(tok))
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1342 return ERRORTOKEN;
dc132cc39573 Add checking for inconsistent tab usage
Guido van Rossum <guido@python.org>
parents: 7324
diff changeset
1343 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1344 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1345 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1346 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1347
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1348 tok->start = tok->cur;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1349
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1350 /* Return pending indents/dedents */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1351 if (tok->pendin != 0) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1352 if (tok->pendin < 0) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1353 tok->pendin++;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1354 return DEDENT;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1355 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1356 else {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1357 tok->pendin--;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1358 return INDENT;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1359 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1360 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1361
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1362 again:
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1363 tok->start = NULL;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1364 /* Skip spaces */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1365 do {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1366 c = tok_nextc(tok);
2801
e911f2fb179c ignore control-l in whitespace
Guido van Rossum <guido@python.org>
parents: 2732
diff changeset
1367 } while (c == ' ' || c == '\t' || c == '\014');
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1368
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1369 /* Set start of current token */
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1370 tok->start = tok->cur - 1;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1371
37704
bb3f9076d079 Make 'python -tt' the default, meaning Python won't allow mixing tabs and
Thomas Wouters <thomas@python.org>
parents: 36809
diff changeset
1372 /* Skip comment */
bb3f9076d079 Make 'python -tt' the default, meaning Python won't allow mixing tabs and
Thomas Wouters <thomas@python.org>
parents: 36809
diff changeset
1373 if (c == '#')
12142
43c36f63f867 Fix by Eric Raymond: make the code that looks for various bits of
Guido van Rossum <guido@python.org>
parents: 11971
diff changeset
1374 while (c != EOF && c != '\n')
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1375 c = tok_nextc(tok);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1376
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1377 /* Check for EOF and errors now */
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1378 if (c == EOF) {
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1379 return tok->done == E_EOF ? ENDMARKER : ERRORTOKEN;
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1380 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1381
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1382 /* Identifier (most frequent token!) */
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1383 nonascii = 0;
42013
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
1384 if (is_potential_identifier_start(c)) {
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1385 /* Process b"", r"" and br"" */
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1386 if (c == 'b' || c == 'B') {
5341
67e5237ce141 Now that the string-sig has settled on r"obin" strings, restrict the
Guido van Rossum <guido@python.org>
parents: 5311
diff changeset
1387 c = tok_nextc(tok);
67e5237ce141 Now that the string-sig has settled on r"obin" strings, restrict the
Guido van Rossum <guido@python.org>
parents: 5311
diff changeset
1388 if (c == '"' || c == '\'')
67e5237ce141 Now that the string-sig has settled on r"obin" strings, restrict the
Guido van Rossum <guido@python.org>
parents: 5311
diff changeset
1389 goto letter_quote;
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1390 }
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1391 if (c == 'r' || c == 'R') {
40894
0ea24f4f2467 Bytes literal.
Thomas Wouters <thomas@python.org>
parents: 40644
diff changeset
1392 c = tok_nextc(tok);
0ea24f4f2467 Bytes literal.
Thomas Wouters <thomas@python.org>
parents: 40644
diff changeset
1393 if (c == '"' || c == '\'')
0ea24f4f2467 Bytes literal.
Thomas Wouters <thomas@python.org>
parents: 40644
diff changeset
1394 goto letter_quote;
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1395 }
42013
cd125fe83051 Make identifiers str (not str8) objects throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 41970
diff changeset
1396 while (is_potential_identifier_char(c)) {
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1397 if (c >= 128)
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1398 nonascii = 1;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1399 c = tok_nextc(tok);
5265
3c1788ab9761 Support for alternative string quotes (a"xx", b"xx", c"xx", ...).
Guido van Rossum <guido@python.org>
parents: 4839
diff changeset
1400 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1401 tok_backup(tok, c);
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1402 if (nonascii &&
42597
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1403 !verify_identifier(tok->start, tok->cur)) {
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1404 tok->done = E_IDENTIFIER;
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1405 return ERRORTOKEN;
20477cbfd996 Implement PEP 3131. Add isidentifier to str.
Martin v. Löwis <martin@v.loewis.de>
parents: 42551
diff changeset
1406 }
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1407 *p_start = tok->start;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1408 *p_end = tok->cur;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1409 return NAME;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1410 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1411
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1412 /* Newline */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1413 if (c == '\n') {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1414 tok->atbol = 1;
1284
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1415 if (blankline || tok->level > 0)
326
ff0862e3d63a Completely ignore lines with only a newline token on them, except
Guido van Rossum <guido@python.org>
parents: 293
diff changeset
1416 goto nextline;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1417 *p_start = tok->start;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1418 *p_end = tok->cur - 1; /* Leave '\n' out of the string */
25098
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
1419 tok->cont_line = 0;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1420 return NEWLINE;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1421 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1422
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1423 /* Period or number starting with period? */
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1424 if (c == '.') {
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1425 c = tok_nextc(tok);
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1426 if (isdigit(c)) {
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1427 goto fraction;
41230
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1428 } else if (c == '.') {
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1429 c = tok_nextc(tok);
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1430 if (c == '.') {
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1431 *p_start = tok->start;
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1432 *p_end = tok->cur;
41230
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1433 return ELLIPSIS;
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1434 } else {
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1435 tok_backup(tok, c);
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1436 }
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1437 tok_backup(tok, '.');
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1438 } else {
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1439 tok_backup(tok, c);
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1440 }
41230
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1441 *p_start = tok->start;
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1442 *p_end = tok->cur;
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis.
Georg Brandl <georg@python.org>
parents: 40894
diff changeset
1443 return DOT;
410
fd061dae9e57 Added shift and mask ops.
Guido van Rossum <guido@python.org>
parents: 400
diff changeset
1444 }
3333
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1445
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1446 /* Number */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1447 if (isdigit(c)) {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1448 if (c == '0') {
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1449 /* Hex, octal or binary -- maybe. */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1450 c = tok_nextc(tok);
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1451 if (c == '.')
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1452 goto fraction;
3364
bc17e8313ea0 use only j for imaginary constants
Guido van Rossum <guido@python.org>
parents: 3333
diff changeset
1453 if (c == 'j' || c == 'J')
3333
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1454 goto imaginary;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1455 if (c == 'x' || c == 'X') {
44782
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1456
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1457 /* Hex */
44782
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1458 c = tok_nextc(tok);
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1459 if (!isxdigit(c)) {
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1460 tok->done = E_TOKEN;
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1461 tok_backup(tok, c);
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1462 return ERRORTOKEN;
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1463 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1464 do {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1465 c = tok_nextc(tok);
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1466 } while (isxdigit(c));
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1467 }
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1468 else if (c == 'o' || c == 'O') {
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1469 /* Octal */
44782
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1470 c = tok_nextc(tok);
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 46161
diff changeset
1471 if (c < '0' || c >= '8') {
44782
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1472 tok->done = E_TOKEN;
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1473 tok_backup(tok, c);
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1474 return ERRORTOKEN;
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1475 }
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1476 do {
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1477 c = tok_nextc(tok);
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1478 } while ('0' <= c && c < '8');
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1479 }
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1480 else if (c == 'b' || c == 'B') {
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1481 /* Binary */
44782
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1482 c = tok_nextc(tok);
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1483 if (c != '0' && c != '1') {
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1484 tok->done = E_TOKEN;
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1485 tok_backup(tok, c);
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1486 return ERRORTOKEN;
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from
Georg Brandl <georg@python.org>
parents: 44442
diff changeset
1487 }
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1488 do {
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1489 c = tok_nextc(tok);
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1490 } while (c == '0' || c == '1');
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1491 }
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1492 else {
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1493 int nonzero = 0;
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1494 /* maybe old-style octal; c is first char of it */
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1495 /* in any case, allow '0' as a literal */
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1496 while (c == '0')
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1497 c = tok_nextc(tok);
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1498 while (isdigit(c)) {
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1499 nonzero = 1;
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1500 c = tok_nextc(tok);
19621
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1501 }
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1502 if (c == '.')
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1503 goto fraction;
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1504 else if (c == 'e' || c == 'E')
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1505 goto exponent;
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1506 else if (c == 'j' || c == 'J')
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1507 goto imaginary;
42084
229c28cb9afb Merged revisions 55817-55961 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 42071
diff changeset
1508 else if (nonzero) {
19621
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1509 tok->done = E_TOKEN;
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1510 tok_backup(tok, c);
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1511 return ERRORTOKEN;
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1512 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1513 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1514 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1515 else {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1516 /* Decimal */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1517 do {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1518 c = tok_nextc(tok);
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1519 } while (isdigit(c));
40644
1c26505c1de6 Rip out 'long' and 'L'-suffixed integer literals.
Guido van Rossum <guido@python.org>
parents: 40529
diff changeset
1520 {
19501
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1521 /* Accept floating point numbers. */
236
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1522 if (c == '.') {
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1523 fraction:
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1524 /* Fraction */
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1525 do {
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1526 c = tok_nextc(tok);
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1527 } while (isdigit(c));
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1528 }
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1529 if (c == 'e' || c == 'E') {
19621
0bec699783d0 SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents: 19501
diff changeset
1530 exponent:
236
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1531 /* Exponent part */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1532 c = tok_nextc(tok);
236
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1533 if (c == '+' || c == '-')
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1534 c = tok_nextc(tok);
19501
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1535 if (!isdigit(c)) {
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1536 tok->done = E_TOKEN;
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1537 tok_backup(tok, c);
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1538 return ERRORTOKEN;
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1539 }
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1540 do {
236
0a92b6a1b9c0 Added recognition of 'l' or 'L' as long integer suffix
Guido van Rossum <guido@python.org>
parents: 145
diff changeset
1541 c = tok_nextc(tok);
19501
ce4f1e98ba7e SF bug [#455775] float parsing discrepancy.
Tim Peters <tim.peters@gmail.com>
parents: 19051
diff changeset
1542 } while (isdigit(c));
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1543 }
3364
bc17e8313ea0 use only j for imaginary constants
Guido van Rossum <guido@python.org>
parents: 3333
diff changeset
1544 if (c == 'j' || c == 'J')
3333
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1545 /* Imaginary part */
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1546 imaginary:
4b5bff95804e changes for pow(**) and complex
Guido van Rossum <guido@python.org>
parents: 3116
diff changeset
1547 c = tok_nextc(tok);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1548 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1549 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1550 tok_backup(tok, c);
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1551 *p_start = tok->start;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1552 *p_end = tok->cur;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1553 return NUMBER;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1554 }
5265
3c1788ab9761 Support for alternative string quotes (a"xx", b"xx", c"xx", ...).
Guido van Rossum <guido@python.org>
parents: 4839
diff changeset
1555
3c1788ab9761 Support for alternative string quotes (a"xx", b"xx", c"xx", ...).
Guido van Rossum <guido@python.org>
parents: 4839
diff changeset
1556 letter_quote:
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1557 /* String */
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1558 if (c == '\'' || c == '"') {
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1559 int quote = c;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1560 int quote_size = 1; /* 1 or 3 */
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1561 int end_quote_size = 0;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1562
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1563 /* Find the quote size and start of string */
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1564 c = tok_nextc(tok);
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1565 if (c == quote) {
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1566 c = tok_nextc(tok);
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1567 if (c == quote)
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1568 quote_size = 3;
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1569 else
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1570 end_quote_size = 1; /* empty string found */
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1571 }
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1572 if (c != quote)
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1573 tok_backup(tok, c);
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1574
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1575 /* Get rest of string */
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1576 while (end_quote_size != quote_size) {
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1577 c = tok_nextc(tok);
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1578 if (c == EOF) {
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1579 if (quote_size == 3)
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1580 tok->done = E_EOFS;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1581 else
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1582 tok->done = E_EOLS;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1583 tok->cur = tok->inp;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1584 return ERRORTOKEN;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1585 }
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1586 if (quote_size == 1 && c == '\n') {
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1587 tok->done = E_EOLS;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1588 tok->cur = tok->inp;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1589 return ERRORTOKEN;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1590 }
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1591 if (c == quote)
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1592 end_quote_size += 1;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1593 else {
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1594 end_quote_size = 0;
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1595 if (c == '\\')
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1596 c = tok_nextc(tok); /* skip escaped char */
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1597 }
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1598 }
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1599
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1600 *p_start = tok->start;
1364
132602d854d8 Changes to accept double-quoted strings on input.
Guido van Rossum <guido@python.org>
parents: 1284
diff changeset
1601 *p_end = tok->cur;
132602d854d8 Changes to accept double-quoted strings on input.
Guido van Rossum <guido@python.org>
parents: 1284
diff changeset
1602 return STRING;
132602d854d8 Changes to accept double-quoted strings on input.
Guido van Rossum <guido@python.org>
parents: 1284
diff changeset
1603 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1604
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1605 /* Line continuation */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1606 if (c == '\\') {
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1607 c = tok_nextc(tok);
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1608 if (c != '\n') {
34664
3785610ff763 Patch #802188: better parser error message for non-EOL following line cont.
Martin v. Löwis <martin@v.loewis.de>
parents: 32961
diff changeset
1609 tok->done = E_LINECONT;
1282
e0464e7115ca * pythonrun.c: Print exception type+arg *after* stack trace instead of
Guido van Rossum <guido@python.org>
parents: 1257
diff changeset
1610 tok->cur = tok->inp;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1611 return ERRORTOKEN;
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1612 }
25098
c02a668488d0 Ignore encoding declarations inside strings. Fixes #603509.
Martin v. Löwis <martin@v.loewis.de>
parents: 24867
diff changeset
1613 tok->cont_line = 1;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1614 goto again; /* Read next line */
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1615 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1616
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1617 /* Check for two-character token */
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1618 {
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1619 int c2 = tok_nextc(tok);
5379
1c9a84e0981d Another directory quickly renamed.
Guido van Rossum <guido@python.org>
parents: 5341
diff changeset
1620 int token = PyToken_TwoChars(c, c2);
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1621 if (token != OP) {
14163
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1622 int c3 = tok_nextc(tok);
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1623 int token3 = PyToken_ThreeChars(c, c2, c3);
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1624 if (token3 != OP) {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1625 token = token3;
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1626 } else {
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1627 tok_backup(tok, c3);
1b9fd0cbd914 Support for three-token characters (**=, >>=, <<=) which was written by
Thomas Wouters <thomas@python.org>
parents: 13659
diff changeset
1628 }
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1629 *p_start = tok->start;
400
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1630 *p_end = tok->cur;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1631 return token;
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1632 }
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1633 tok_backup(tok, c2);
d189a2f681a6 Added 2-char tokens and new versions of comparisons
Guido van Rossum <guido@python.org>
parents: 326
diff changeset
1634 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1635
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1636 /* Keep track of parentheses nesting level */
1284
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1637 switch (c) {
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1638 case '(':
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1639 case '[':
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1640 case '{':
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1641 tok->level++;
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1642 break;
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1643 case ')':
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1644 case ']':
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1645 case '}':
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1646 tok->level--;
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1647 break;
dbce72afe261 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
Guido van Rossum <guido@python.org>
parents: 1282
diff changeset
1648 }
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37704
diff changeset
1649
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1650 /* Punctuation character */
1835
d799889c01e3 * Parser/tokenizer.c: backup over illegal newline in string
Guido van Rossum <guido@python.org>
parents: 1364
diff changeset
1651 *p_start = tok->start;
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1652 *p_end = tok->cur;
5379
1c9a84e0981d Another directory quickly renamed.
Guido van Rossum <guido@python.org>
parents: 5341
diff changeset
1653 return PyToken_OneChar(c);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1654 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1655
24539
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1656 int
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1657 PyTokenizer_Get(struct tok_state *tok, char **p_start, char **p_end)
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1658 {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1659 int result = tok_get(tok, p_start, p_end);
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1660 if (tok->decoding_erred) {
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1661 result = ERRORTOKEN;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1662 tok->done = E_DECODE;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1663 }
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1664 return result;
06f1e2fd05bc Patch #534304: Implement phase 1 of PEP 263.
Martin v. Löwis <martin@v.loewis.de>
parents: 23029
diff changeset
1665 }
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1666
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1667 /* Get -*- encoding -*- from a Python file.
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1668
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1669 PyTokenizer_FindEncoding returns NULL when it can't find the encoding in
43879
5590157f4fc6 Cleanup of tokenizer.c.
Guido van Rossum <guido@python.org>
parents: 43873
diff changeset
1670 the first or second line of the file (in which case the encoding
43609
d378acdc30ee Fix PyTokenizer_FindEncoding() for OS X 10.4. Turns out that seeking to the
Brett Cannon <bcannon@gmail.com>
parents: 43607
diff changeset
1671 should be assumed to be PyUnicode_GetDefaultEncoding()).
d378acdc30ee Fix PyTokenizer_FindEncoding() for OS X 10.4. Turns out that seeking to the
Brett Cannon <bcannon@gmail.com>
parents: 43607
diff changeset
1672
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1673 The char * returned is malloc'ed via PyMem_MALLOC() and thus must be freed
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1674 by the caller.
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1675 */
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1676 char *
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1677 PyTokenizer_FindEncoding(int fd)
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1678 {
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1679 struct tok_state *tok;
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1680 FILE *fp;
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1681 char *p_start =NULL , *p_end =NULL , *encoding = NULL;
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1682
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1683 fd = dup(fd);
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1684 if (fd < 0) {
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1685 return NULL;
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1686 }
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1687 fp = fdopen(fd, "r");
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1688 if (fp == NULL) {
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1689 return NULL;
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1690 }
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1691 tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL);
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1692 if (tok == NULL) {
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1693 fclose(fp);
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1694 return NULL;
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1695 }
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1696 while (tok->lineno < 2 && tok->done == E_OK) {
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1697 PyTokenizer_Get(tok, &p_start, &p_end);
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1698 }
43632
0067a6ab4854 Issue 1267, continued.
Guido van Rossum <guido@python.org>
parents: 43622
diff changeset
1699 fclose(fp);
43607
7593436058dd Plug a memory leak where a struct tok_state was not being freed.
Brett Cannon <bcannon@gmail.com>
parents: 43605
diff changeset
1700 if (tok->encoding) {
43622
8fe226931ee7 Make sure the malloc'ed string has space for the null byte.
Brett Cannon <bcannon@gmail.com>
parents: 43609
diff changeset
1701 encoding = (char *)PyMem_MALLOC(strlen(tok->encoding) + 1);
49020
3fa066d2a397 #3773: Check for errors around the use of PyTokenizer_FindEncoding().
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49007
diff changeset
1702 if (encoding)
3fa066d2a397 #3773: Check for errors around the use of PyTokenizer_FindEncoding().
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents: 49007
diff changeset
1703 strcpy(encoding, tok->encoding);
43607
7593436058dd Plug a memory leak where a struct tok_state was not being freed.
Brett Cannon <bcannon@gmail.com>
parents: 43605
diff changeset
1704 }
7593436058dd Plug a memory leak where a struct tok_state was not being freed.
Brett Cannon <bcannon@gmail.com>
parents: 43605
diff changeset
1705 PyTokenizer_Free(tok);
7593436058dd Plug a memory leak where a struct tok_state was not being freed.
Brett Cannon <bcannon@gmail.com>
parents: 43605
diff changeset
1706 return encoding;
43605
a2bb76d11d66 Patch 1267 by Christian Heimes.
Guido van Rossum <guido@python.org>
parents: 43523
diff changeset
1707 }
43292
876b7ef4775d Merged revisions 57778-58052 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 43225
diff changeset
1708
4839
4567e158d20c Rename DEBUG macro to Py_DEBUG
Guido van Rossum <guido@python.org>
parents: 4577
diff changeset
1709 #ifdef Py_DEBUG
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1710
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1711 void
13659
e78153e179fb Mass ANSIfication.
Thomas Wouters <thomas@python.org>
parents: 13539
diff changeset
1712 tok_dump(int type, char *start, char *end)
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1713 {
5379
1c9a84e0981d Another directory quickly renamed.
Guido van Rossum <guido@python.org>
parents: 5341
diff changeset
1714 printf("%s", _PyParser_TokenNames[type]);
6
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1715 if (type == NAME || type == NUMBER || type == STRING || type == OP)
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1716 printf("(%.*s)", (int)(end - start), start);
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1717 }
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1718
fc6fcd7df4f7 Initial revision
Guido van Rossum <guido@python.org>
parents:
diff changeset
1719 #endif