|
3198
|
1 % libparser.tex
|
|
|
2 %
|
|
|
3 % Introductory documentation for the new parser built-in module.
|
|
|
4 %
|
|
|
5 % Copyright 1995 Virginia Polytechnic Institute and State University
|
|
|
6 % and Fred L. Drake, Jr. This copyright notice must be distributed on
|
|
|
7 % all copies, but this document otherwise may be distributed as part
|
|
|
8 % of the Python distribution. No fee may be charged for this document
|
|
|
9 % in any representation, either on paper or electronically. This
|
|
|
10 % restriction does not affect other elements in a distributed package
|
|
|
11 % in any way.
|
|
|
12 %
|
|
|
13
|
|
|
14 \section{Built-in Module \sectcode{parser}}
|
|
|
15 \bimodindex{parser}
|
|
|
16
|
|
|
17
|
|
|
18 % ==== 2. ====
|
|
|
19 % Give a short overview of what the module does.
|
|
|
20 % If it is platform specific, mention this.
|
|
|
21 % Mention other important restrictions or general operating principles.
|
|
|
22
|
|
|
23 The \code{parser} module provides an interface to Python's internal
|
|
|
24 parser and byte-code compiler. The primary purpose for this interface
|
|
|
25 is to allow Python code to edit the parse tree of a Python expression
|
|
|
26 and create executable code from this. This can be better than trying
|
|
|
27 to parse and modify an arbitrary Python code fragment as a string, and
|
|
|
28 ensures that parsing is performed in a manner identical to the code
|
|
|
29 forming the application. It's also faster.
|
|
|
30
|
|
|
31 There are a few things to note about this module which are important
|
|
|
32 to making use of the data structures created. This is not a tutorial
|
|
|
33 on editing the parse trees for Python code.
|
|
|
34
|
|
|
35 Most importantly, a good understanding of the Python grammar processed
|
|
|
36 by the internal parser is required. For full information on the
|
|
|
37 language syntax, refer to the Language Reference. The parser itself
|
|
|
38 is created from a grammar specification defined in the file
|
|
|
39 \code{Grammar/Grammar} in the standard Python distribution. The parse
|
|
|
40 trees stored in the ``AST objects'' created by this module are the
|
|
|
41 actual output from the internal parser when created by the
|
|
|
42 \code{expr()} or \code{suite()} functions, described below. The AST
|
|
|
43 objects created by \code{tuple2ast()} faithfully simulate those
|
|
|
44 structures.
|
|
|
45
|
|
|
46 Each element of the tuples returned by \code{ast2tuple()} has a simple
|
|
|
47 form. Tuples representing non-terminal elements in the grammar always
|
|
|
48 have a length greater than one. The first element is an integer which
|
|
|
49 identifies a production in the grammar. These integers are given
|
|
|
50 symbolic names in the C header file \code{Include/graminit.h} and the
|
|
|
51 Python module \code{Lib/symbol.py}. Each additional element of the
|
|
|
52 tuple represents a component of the production as recognized in the
|
|
|
53 input string: these are always tuples which have the same form as the
|
|
|
54 parent. An important aspect of this structure which should be noted
|
|
|
55 is that keywords used to identify the parent node type, such as the
|
|
|
56 keyword \code{if} in an \emph{if\_stmt}, are included in the node tree
|
|
|
57 without any special treatment. For example, the \code{if} keyword is
|
|
|
58 represented by the tuple \code{(1, 'if')}, where \code{1} is the
|
|
|
59 numeric value associated with all \code{NAME} elements, including
|
|
|
60 variable and function names defined by the user.
|
|
|
61
|
|
|
62 Terminal elements are represented in much the same way, but without
|
|
|
63 any child elements and the addition of the source text which was
|
|
|
64 identified. The example of the \code{if} keyword above is
|
|
|
65 representative. The various types of terminal symbols are defined in
|
|
|
66 the C header file \code{Include/token.h} and the Python module
|
|
|
67 \code{Lib/token.py}.
|
|
|
68
|
|
|
69 The AST objects are not actually required to support the functionality
|
|
|
70 of this module, but are provided for three purposes: to allow an
|
|
|
71 application to amortize the cost of processing complex parse trees, to
|
|
|
72 provide a parse tree representation which conserves memory space when
|
|
|
73 compared to the Python tuple representation, and to ease the creation
|
|
|
74 of additional modules in C which manipulate parse trees. A simple
|
|
|
75 ``wrapper'' module may be created in Python if desired to hide the use
|
|
|
76 of AST objects.
|
|
|
77
|
|
|
78
|
|
|
79 % ==== 3. ====
|
|
|
80 % List the public functions defined by the module. Begin with a
|
|
|
81 % standard phrase. You may also list the exceptions and other data
|
|
|
82 % items defined in the module, insofar as they are important for the
|
|
|
83 % user.
|
|
|
84
|
|
|
85 The \code{parser} module defines the following functions:
|
|
|
86
|
|
|
87 % ---- 3.1. ----
|
|
|
88 % Redefine the ``indexsubitem'' macro to point to this module
|
|
|
89 % (alternatively, you can put this at the top of the file):
|
|
|
90
|
|
|
91 \renewcommand{\indexsubitem}{(in module parser)}
|
|
|
92
|
|
|
93 % ---- 3.2. ----
|
|
|
94 % For each function, use a ``funcdesc'' block. This has exactly two
|
|
|
95 % parameters (each parameters is contained in a set of curly braces):
|
|
|
96 % the first parameter is the function name (this automatically
|
|
|
97 % generates an index entry); the second parameter is the function's
|
|
|
98 % argument list. If there are no arguments, use an empty pair of
|
|
|
99 % curly braces. If there is more than one argument, separate the
|
|
|
100 % arguments with backslash-comma. Optional parts of the parameter
|
|
|
101 % list are contained in \optional{...} (this generates a set of square
|
|
|
102 % brackets around its parameter). Arguments are automatically set in
|
|
|
103 % italics in the parameter list. Each argument should be mentioned at
|
|
|
104 % least once in the description; each usage (even inside \code{...})
|
|
|
105 % should be enclosed in \var{...}.
|
|
|
106
|
|
|
107 \begin{funcdesc}{ast2tuple}{ast}
|
|
|
108 This function accepts an AST object from the caller in
|
|
|
109 \code{\var{ast}} and returns a Python tuple representing the
|
|
|
110 equivelent parse tree. The resulting tuple representation can be used
|
|
|
111 for inspection or the creation of a new parse tree in tuple form.
|
|
|
112 This function does not fail so long as memory is available to build
|
|
|
113 the tuple representation.
|
|
|
114 \end{funcdesc}
|
|
|
115
|
|
|
116
|
|
|
117 \begin{funcdesc}{compileast}{ast\optional{\, filename \code{= '<ast>'}}}
|
|
|
118 The Python byte compiler can be invoked on an AST object to produce
|
|
|
119 code objects which can be used as part of an \code{exec} statement or
|
|
|
120 a call to the built-in \code{eval()} function. This function provides
|
|
|
121 the interface to the compiler, passing the internal parse tree from
|
|
|
122 \code{\var{ast}} to the parser, using the source file name specified
|
|
|
123 by the \code{\var{filename}} parameter. The default value supplied
|
|
|
124 for \code{\var{filename}} indicates that the source was an AST object.
|
|
|
125 \end{funcdesc}
|
|
|
126
|
|
|
127
|
|
|
128 \begin{funcdesc}{expr}{string}
|
|
|
129 The \code{expr()} function parses the parameter \code{\var{string}}
|
|
|
130 as if it were an input to \code{compile(\var{string}, 'eval')}. If
|
|
|
131 the parse succeeds, an AST object is created to hold the internal
|
|
|
132 parse tree representation, otherwise an appropriate exception is
|
|
|
133 thrown.
|
|
|
134 \end{funcdesc}
|
|
|
135
|
|
|
136
|
|
|
137 \begin{funcdesc}{isexpr}{ast}
|
|
|
138 When \code{\var{ast}} represents an \code{'eval'} form, this function
|
|
|
139 returns a true value (\code{1}), otherwise it returns false
|
|
|
140 (\code{0}). This is useful, since code objects normally cannot be
|
|
|
141 queried for this information using existing built-in functions. Note
|
|
|
142 that the code objects created by \code{compileast()} cannot be queried
|
|
|
143 like this either, and are identical to those created by the built-in
|
|
|
144 \code{compile()} function.
|
|
|
145 \end{funcdesc}
|
|
|
146
|
|
|
147
|
|
|
148 \begin{funcdesc}{issuite}{ast}
|
|
|
149 This function mirrors \code{isexpr()} in that it reports whether an
|
|
|
150 AST object represents a suite of statements. It is not safe to assume
|
|
|
151 that this function is equivelent to \code{not isexpr(\var{ast})}, as
|
|
|
152 additional syntactic fragments may be supported in the future.
|
|
|
153 \end{funcdesc}
|
|
|
154
|
|
|
155
|
|
|
156 \begin{funcdesc}{suite}{string}
|
|
|
157 The \code{suite()} function parses the parameter \code{\var{string}}
|
|
|
158 as if it were an input to \code{compile(\var{string}, 'exec')}. If
|
|
|
159 the parse succeeds, an AST object is created to hold the internal
|
|
|
160 parse tree representation, otherwise an appropriate exception is
|
|
|
161 thrown.
|
|
|
162 \end{funcdesc}
|
|
|
163
|
|
|
164
|
|
|
165 \begin{funcdesc}{tuple2ast}{tuple}
|
|
|
166 This function accepts a parse tree represented as a tuple and builds
|
|
|
167 an internal representation if possible. If it can validate that the
|
|
|
168 tree conforms to the Python syntax and all nodes are valid node types
|
|
|
169 in the host version of Python, an AST object is created from the
|
|
|
170 internal representation and returned to the called. If there is a
|
|
|
171 problem creating the internal representation, or if the tree cannot be
|
|
|
172 validated, a \code{ParserError} exception is thrown. An AST object
|
|
|
173 created this way should not be assumed to compile correctly; normal
|
|
|
174 exceptions thrown by compilation may still be initiated when the AST
|
|
|
175 object is passed to \code{compileast()}. This will normally indicate
|
|
|
176 problems not related to syntax (such as a \code{MemoryError}
|
|
|
177 exception).
|
|
|
178 \end{funcdesc}
|
|
|
179
|
|
|
180
|
|
|
181 % --- 3.4. ---
|
|
|
182 % Exceptions are described using a ``excdesc'' block. This has only
|
|
|
183 % one parameter: the exception name.
|
|
|
184
|
|
|
185 \subsection{Exceptions and Error Handling}
|
|
|
186
|
|
|
187 The parser module defines a single exception, but may also pass other
|
|
|
188 built-in exceptions from other portions of the Python runtime
|
|
|
189 environment. See each function for information about the exceptions
|
|
|
190 it can raise.
|
|
|
191
|
|
|
192 \begin{excdesc}{ParserError}
|
|
|
193 Exception raised when a failure occurs within the parser module. This
|
|
|
194 is generally produced for validation failures rather than the built in
|
|
|
195 \code{SyntaxError} thrown during normal parsing.
|
|
|
196 The exception argument is either a string describing the reason of the
|
|
|
197 failure or a tuple containing a tuple causing the failure from a parse
|
|
|
198 tree passed to \code{tuple2ast()} and an explanatory string. Calls to
|
|
|
199 \code{tuple2ast()} need to be able to handle either type of exception,
|
|
|
200 while calls to other functions in the module will only need to be
|
|
|
201 aware of the simple string values.
|
|
|
202 \end{excdesc}
|
|
|
203
|
|
|
204 Note that the functions \code{compileast()}, \code{expr()}, and
|
|
|
205 \code{suite()} may throw exceptions which are normally thrown by the
|
|
|
206 parsing and compilation process. These include the built in
|
|
|
207 exceptions \code{MemoryError}, \code{OverflowError},
|
|
|
208 \code{SyntaxError}, and \code{SystemError}. In these cases, these
|
|
|
209 exceptions carry all the meaning normally associated with them. Refer
|
|
|
210 to the descriptions of each function for detailed information.
|
|
|
211
|
|
|
212 % ---- 3.5. ----
|
|
|
213 % There is no standard block type for classes. I generally use
|
|
|
214 % ``funcdesc'' blocks, since class instantiation looks very much like
|
|
|
215 % a function call.
|
|
|
216
|
|
|
217
|
|
|
218 % ==== 4. ====
|
|
|
219 % Now is probably a good time for a complete example. (Alternatively,
|
|
|
220 % an example giving the flavor of the module may be given before the
|
|
|
221 % detailed list of functions.)
|
|
|
222
|
|
|
223 \subsection{Example}
|
|
|
224
|
|
|
225 A simple example:
|
|
|
226
|
|
|
227 \begin{verbatim}
|
|
|
228 >>> import parser
|
|
|
229 >>> ast = parser.expr('a + 5')
|
|
|
230 >>> code = parser.compileast(ast)
|
|
|
231 >>> a = 5
|
|
|
232 >>> eval(code)
|
|
|
233 10
|
|
|
234 \end{verbatim}
|
|
|
235
|
|
|
236
|
|
|
237 \subsection{AST Objects}
|
|
|
238
|
|
|
239 AST objects (returned by \code{expr()}, \code{suite()}, and
|
|
|
240 \code{tuple2ast()}, described above) have no methods of their own.
|
|
|
241 Some of the functions defined which accept an AST object as their
|
|
|
242 first argument may change to object methods in the future.
|
|
|
243
|
|
|
244 Ordered and equality comparisons are supported between AST objects.
|
|
|
245
|
|
|
246 \renewcommand{\indexsubitem}{(ast method)}
|
|
|
247
|
|
|
248 %\begin{funcdesc}{empty}{}
|
|
|
249 %Empty the can into the trash.
|
|
|
250 %\end{funcdesc}
|
