Mercurial > cpython-lru_cache-weakref
annotate Doc/libparser.tex @ 7050:90edcb38fb00 legacy-trunk
Added \label{} to allow module references.
Added some seealso sections.
| author | Fred Drake <fdrake@acm.org> |
|---|---|
| date | Fri, 09 Jan 1998 22:24:14 +0000 |
| parents | 0f2498b847bf |
| children | 9dde07a593d8 |
| rev | line source |
|---|---|
| 3198 | 1 % libparser.tex |
| 2 % | |
| 3 % Copyright 1995 Virginia Polytechnic Institute and State University | |
| 4 % and Fred L. Drake, Jr. This copyright notice must be distributed on | |
| 5 % all copies, but this document otherwise may be distributed as part | |
| 6 % of the Python distribution. No fee may be charged for this document | |
| 7 % in any representation, either on paper or electronically. This | |
| 8 % restriction does not affect other elements in a distributed package | |
| 9 % in any way. | |
| 10 % | |
| 11 | |
| 12 \section{Built-in Module \sectcode{parser}} | |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
13 \label{module-parser} |
| 3198 | 14 \bimodindex{parser} |
| 15 | |
| 16 The \code{parser} module provides an interface to Python's internal | |
| 17 parser and byte-code compiler. The primary purpose for this interface | |
| 18 is to allow Python code to edit the parse tree of a Python expression | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
19 and create executable code from this. This is better than trying |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
20 to parse and modify an arbitrary Python code fragment as a string |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
21 because parsing is performed in a manner identical to the code |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
22 forming the application. It is also faster. |
| 3198 | 23 |
| 24 There are a few things to note about this module which are important | |
| 25 to making use of the data structures created. This is not a tutorial | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
26 on editing the parse trees for Python code, but some examples of using |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
27 the \code{parser} module are presented. |
| 3198 | 28 |
| 29 Most importantly, a good understanding of the Python grammar processed | |
| 30 by the internal parser is required. For full information on the | |
| 31 language syntax, refer to the Language Reference. The parser itself | |
| 32 is created from a grammar specification defined in the file | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
33 \file{Grammar/Grammar} in the standard Python distribution. The parse |
| 3198 | 34 trees stored in the ``AST objects'' created by this module are the |
| 35 actual output from the internal parser when created by the | |
| 36 \code{expr()} or \code{suite()} functions, described below. The AST | |
| 4084 | 37 objects created by \code{sequence2ast()} faithfully simulate those |
| 38 structures. Be aware that the values of the sequences which are | |
| 39 considered ``correct'' will vary from one version of Python to another | |
| 40 as the formal grammar for the language is revised. However, | |
| 41 transporting code from one Python version to another as source text | |
| 42 will always allow correct parse trees to be created in the target | |
| 43 version, with the only restriction being that migrating to an older | |
| 44 version of the interpreter will not support more recent language | |
| 45 constructs. The parse trees are not typically compatible from one | |
| 46 version to another, whereas source code has always been | |
| 47 forward-compatible. | |
| 3198 | 48 |
| 4084 | 49 Each element of the sequences returned by \code{ast2list} or |
| 50 \code{ast2tuple()} has a simple form. Sequences representing | |
| 51 non-terminal elements in the grammar always have a length greater than | |
| 52 one. The first element is an integer which identifies a production in | |
| 53 the grammar. These integers are given symbolic names in the C header | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
54 file \file{Include/graminit.h} and the Python module |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
55 \code{symbol}. Each additional element of the sequence represents |
| 4084 | 56 a component of the production as recognized in the input string: these |
| 57 are always sequences which have the same form as the parent. An | |
| 58 important aspect of this structure which should be noted is that | |
| 59 keywords used to identify the parent node type, such as the keyword | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
60 \code{if} in an \code{if_stmt}, are included in the node tree without |
| 4084 | 61 any special treatment. For example, the \code{if} keyword is |
| 3198 | 62 represented by the tuple \code{(1, 'if')}, where \code{1} is the |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
63 numeric value associated with all \code{NAME} tokens, including |
| 4084 | 64 variable and function names defined by the user. In an alternate form |
| 65 returned when line number information is requested, the same token | |
| 66 might be represented as \code{(1, 'if', 12)}, where the \code{12} | |
| 67 represents the line number at which the terminal symbol was found. | |
| 3198 | 68 |
| 69 Terminal elements are represented in much the same way, but without | |
| 70 any child elements and the addition of the source text which was | |
| 71 identified. The example of the \code{if} keyword above is | |
| 72 representative. The various types of terminal symbols are defined in | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
73 the C header file \file{Include/token.h} and the Python module |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
74 \code{token}. |
| 3198 | 75 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
76 The AST objects are not required to support the functionality of this |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
77 module, but are provided for three purposes: to allow an application |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
78 to amortize the cost of processing complex parse trees, to provide a |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
79 parse tree representation which conserves memory space when compared |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
80 to the Python list or tuple representation, and to ease the creation |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
81 of additional modules in C which manipulate parse trees. A simple |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
82 ``wrapper'' class may be created in Python to hide the use of AST |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
83 objects; the \code{AST} library module provides a variety of such |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
84 classes. |
| 3198 | 85 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
86 The \code{parser} module defines functions for a few distinct |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
87 purposes. The most important purposes are to create AST objects and |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
88 to convert AST objects to other representations such as parse trees |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
89 and compiled code objects, but there are also functions which serve to |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
90 query the type of parse tree represented by an AST object. |
| 3198 | 91 |
| 92 \renewcommand{\indexsubitem}{(in module parser)} | |
| 93 | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
94 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
95 \subsection{Creating AST Objects} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
96 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
97 AST objects may be created from source code or from a parse tree. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
98 When creating an AST object from source, different functions are used |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
99 to create the \code{'eval'} and \code{'exec'} forms. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
100 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
101 \begin{funcdesc}{expr}{string} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
102 The \code{expr()} function parses the parameter \code{\var{string}} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
103 as if it were an input to \code{compile(\var{string}, 'eval')}. If |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
104 the parse succeeds, an AST object is created to hold the internal |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
105 parse tree representation, otherwise an appropriate exception is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
106 thrown. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
107 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
108 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
109 \begin{funcdesc}{suite}{string} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
110 The \code{suite()} function parses the parameter \code{\var{string}} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
111 as if it were an input to \code{compile(\var{string}, 'exec')}. If |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
112 the parse succeeds, an AST object is created to hold the internal |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
113 parse tree representation, otherwise an appropriate exception is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
114 thrown. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
115 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
116 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
117 \begin{funcdesc}{sequence2ast}{sequence} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
118 This function accepts a parse tree represented as a sequence and |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
119 builds an internal representation if possible. If it can validate |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
120 that the tree conforms to the Python grammar and all nodes are valid |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
121 node types in the host version of Python, an AST object is created |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
122 from the internal representation and returned to the called. If there |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
123 is a problem creating the internal representation, or if the tree |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
124 cannot be validated, a \code{ParserError} exception is thrown. An AST |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
125 object created this way should not be assumed to compile correctly; |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
126 normal exceptions thrown by compilation may still be initiated when |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
127 the AST object is passed to \code{compileast()}. This may indicate |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
128 problems not related to syntax (such as a \code{MemoryError} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
129 exception), but may also be due to constructs such as the result of |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
130 parsing \code{del f(0)}, which escapes the Python parser but is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
131 checked by the bytecode compiler. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
132 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
133 Sequences representing terminal tokens may be represented as either |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
134 two-element lists of the form \code{(1, 'name')} or as three-element |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
135 lists of the form \code{(1, 'name', 56)}. If the third element is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
136 present, it is assumed to be a valid line number. The line number |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
137 may be specified for any subset of the terminal symbols in the input |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
138 tree. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
139 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
140 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
141 \begin{funcdesc}{tuple2ast}{sequence} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
142 This is the same function as \code{sequence2ast()}. This entry point |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
143 is maintained for backward compatibility. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
144 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
145 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
146 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
147 \subsection{Converting AST Objects} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
148 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
149 AST objects, regardless of the input used to create them, may be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
150 converted to parse trees represented as list- or tuple- trees, or may |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
151 be compiled into executable code objects. Parse trees may be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
152 extracted with or without line numbering information. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
153 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
154 \begin{funcdesc}{ast2list}{ast\optional{\, line_info\code{ = 0}}} |
| 4084 | 155 This function accepts an AST object from the caller in |
| 156 \code{\var{ast}} and returns a Python list representing the | |
| 157 equivelent parse tree. The resulting list representation can be used | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
158 for inspection or the creation of a new parse tree in list form. This |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
159 function does not fail so long as memory is available to build the |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
160 list representation. If the parse tree will only be used for |
| 4084 | 161 inspection, \code{ast2tuple()} should be used instead to reduce memory |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
162 consumption and fragmentation. When the list representation is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
163 required, this function is significantly faster than retrieving a |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
164 tuple representation and converting that to nested lists. |
| 4084 | 165 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
166 If \code{\var{line_info}} is true, line number information will be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
167 included for all terminal tokens as a third element of the list |
|
4591
8cdd94a3d02d
(libparser.tex): Added note that the line number for a terminal token
Fred Drake <fdrake@acm.org>
parents:
4286
diff
changeset
|
168 representing the token. Note that the line number provided specifies |
|
8cdd94a3d02d
(libparser.tex): Added note that the line number for a terminal token
Fred Drake <fdrake@acm.org>
parents:
4286
diff
changeset
|
169 the line on which the token \emph{ends\/}. This information is |
|
8cdd94a3d02d
(libparser.tex): Added note that the line number for a terminal token
Fred Drake <fdrake@acm.org>
parents:
4286
diff
changeset
|
170 omitted if the flag is false or omitted. |
| 4084 | 171 \end{funcdesc} |
| 172 | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
173 \begin{funcdesc}{ast2tuple}{ast\optional{\, line_info\code{ = 0}}} |
| 3198 | 174 This function accepts an AST object from the caller in |
| 175 \code{\var{ast}} and returns a Python tuple representing the | |
| 4084 | 176 equivelent parse tree. Other than returning a tuple instead of a |
| 177 list, this function is identical to \code{ast2list()}. | |
| 178 | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
179 If \code{\var{line_info}} is true, line number information will be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
180 included for all terminal tokens as a third element of the list |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
181 representing the token. This information is omitted if the flag is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
182 false or omitted. |
| 3198 | 183 \end{funcdesc} |
| 184 | |
| 4084 | 185 \begin{funcdesc}{compileast}{ast\optional{\, filename\code{ = '<ast>'}}} |
| 3198 | 186 The Python byte compiler can be invoked on an AST object to produce |
| 187 code objects which can be used as part of an \code{exec} statement or | |
| 188 a call to the built-in \code{eval()} function. This function provides | |
| 189 the interface to the compiler, passing the internal parse tree from | |
| 190 \code{\var{ast}} to the parser, using the source file name specified | |
| 191 by the \code{\var{filename}} parameter. The default value supplied | |
| 192 for \code{\var{filename}} indicates that the source was an AST object. | |
| 4084 | 193 |
| 194 Compiling an AST object may result in exceptions related to | |
| 195 compilation; an example would be a \code{SyntaxError} caused by the | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
196 parse tree for \code{del f(0)}: this statement is considered legal |
| 4084 | 197 within the formal grammar for Python but is not a legal language |
| 198 construct. The \code{SyntaxError} raised for this condition is | |
| 199 actually generated by the Python byte-compiler normally, which is why | |
| 200 it can be raised at this point by the \code{parser} module. Most | |
| 201 causes of compilation failure can be diagnosed programmatically by | |
| 202 inspection of the parse tree. | |
| 3198 | 203 \end{funcdesc} |
| 204 | |
| 205 | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
206 \subsection{Queries on AST Objects} |
| 3198 | 207 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
208 Two functions are provided which allow an application to determine if |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
209 an AST was create as an expression or a suite. Neither of these |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
210 functions can be used to determine if an AST was created from source |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
211 code via \code{expr()} or \code{suite()} or from a parse tree via |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
212 \code{sequence2ast()}. |
| 3198 | 213 |
| 214 \begin{funcdesc}{isexpr}{ast} | |
| 215 When \code{\var{ast}} represents an \code{'eval'} form, this function | |
| 216 returns a true value (\code{1}), otherwise it returns false | |
| 217 (\code{0}). This is useful, since code objects normally cannot be | |
| 218 queried for this information using existing built-in functions. Note | |
| 219 that the code objects created by \code{compileast()} cannot be queried | |
| 220 like this either, and are identical to those created by the built-in | |
| 221 \code{compile()} function. | |
| 222 \end{funcdesc} | |
| 223 | |
| 224 | |
| 225 \begin{funcdesc}{issuite}{ast} | |
| 226 This function mirrors \code{isexpr()} in that it reports whether an | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
227 AST object represents an \code{'exec'} form, commonly known as a |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
228 ``suite.'' It is not safe to assume that this function is equivelent |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
229 to \code{not isexpr(\var{ast})}, as additional syntactic fragments may |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
230 be supported in the future. |
| 3198 | 231 \end{funcdesc} |
| 232 | |
| 233 | |
| 234 \subsection{Exceptions and Error Handling} | |
| 235 | |
| 236 The parser module defines a single exception, but may also pass other | |
| 237 built-in exceptions from other portions of the Python runtime | |
| 238 environment. See each function for information about the exceptions | |
| 239 it can raise. | |
| 240 | |
| 241 \begin{excdesc}{ParserError} | |
| 242 Exception raised when a failure occurs within the parser module. This | |
| 243 is generally produced for validation failures rather than the built in | |
| 244 \code{SyntaxError} thrown during normal parsing. | |
| 245 The exception argument is either a string describing the reason of the | |
| 4084 | 246 failure or a tuple containing a sequence causing the failure from a parse |
| 247 tree passed to \code{sequence2ast()} and an explanatory string. Calls to | |
| 248 \code{sequence2ast()} need to be able to handle either type of exception, | |
| 3198 | 249 while calls to other functions in the module will only need to be |
| 250 aware of the simple string values. | |
| 251 \end{excdesc} | |
| 252 | |
| 253 Note that the functions \code{compileast()}, \code{expr()}, and | |
| 254 \code{suite()} may throw exceptions which are normally thrown by the | |
| 255 parsing and compilation process. These include the built in | |
| 256 exceptions \code{MemoryError}, \code{OverflowError}, | |
| 257 \code{SyntaxError}, and \code{SystemError}. In these cases, these | |
| 258 exceptions carry all the meaning normally associated with them. Refer | |
| 259 to the descriptions of each function for detailed information. | |
| 260 | |
| 261 | |
| 4084 | 262 \subsection{AST Objects} |
| 263 | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
264 AST objects returned by \code{expr()}, \code{suite()}, and |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
265 \code{sequence2ast()} have no methods of their own. |
| 4084 | 266 Some of the functions defined which accept an AST object as their |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
267 first argument may change to object methods in the future. The type |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
268 of these objects is available as \code{ASTType} in the module. |
| 4084 | 269 |
| 270 Ordered and equality comparisons are supported between AST objects. | |
| 271 | |
| 272 | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
273 \subsection{Examples} |
|
4728
b218f294b726
(lib<all sorts of stuff>.tex):
Fred Drake <fdrake@acm.org>
parents:
4591
diff
changeset
|
274 \nodename{AST Examples} |
| 3198 | 275 |
| 4084 | 276 The parser modules allows operations to be performed on the parse tree |
| 277 of Python source code before the bytecode is generated, and provides | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
278 for inspection of the parse tree for information gathering purposes. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
279 Two examples are presented. The simple example demonstrates emulation |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
280 of the \code{compile()} built-in function and the complex example |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
281 shows the use of a parse tree for information discovery. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
282 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
283 \subsubsection{Emulation of \sectcode{compile()}} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
284 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
285 While many useful operations may take place between parsing and |
| 4084 | 286 bytecode generation, the simplest operation is to do nothing. For |
| 287 this purpose, using the \code{parser} module to produce an | |
| 288 intermediate data structure is equivelent to the code | |
| 289 | |
| 5818 | 290 \bcode\begin{verbatim} |
| 4084 | 291 >>> code = compile('a + 5', 'eval') |
| 292 >>> a = 5 | |
| 293 >>> eval(code) | |
| 294 10 | |
| 5818 | 295 \end{verbatim}\ecode |
| 296 % | |
| 4084 | 297 The equivelent operation using the \code{parser} module is somewhat |
| 298 longer, and allows the intermediate internal parse tree to be retained | |
| 299 as an AST object: | |
| 3198 | 300 |
| 5818 | 301 \bcode\begin{verbatim} |
| 3198 | 302 >>> import parser |
| 303 >>> ast = parser.expr('a + 5') | |
| 304 >>> code = parser.compileast(ast) | |
| 305 >>> a = 5 | |
| 306 >>> eval(code) | |
| 307 10 | |
| 5818 | 308 \end{verbatim}\ecode |
| 309 % | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
310 An application which needs both AST and code objects can package this |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
311 code into readily available functions: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
312 |
| 5818 | 313 \bcode\begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
314 import parser |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
315 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
316 def load_suite(source_string): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
317 ast = parser.suite(source_string) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
318 code = parser.compileast(ast) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
319 return ast, code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
320 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
321 def load_expression(source_string): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
322 ast = parser.expr(source_string) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
323 code = parser.compileast(ast) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
324 return ast, code |
| 5818 | 325 \end{verbatim}\ecode |
| 326 % | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
327 \subsubsection{Information Discovery} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
328 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
329 Some applications benefit from direct access to the parse tree. The |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
330 remainder of this section demonstrates how the parse tree provides |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
331 access to module documentation defined in docstrings without requiring |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
332 that the code being examined be loaded into a running interpreter via |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
333 \code{import}. This can be very useful for performing analyses of |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
334 untrusted code. |
| 3198 | 335 |
| 4084 | 336 Generally, the example will demonstrate how the parse tree may be |
| 337 traversed to distill interesting information. Two functions and a set | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
338 of classes are developed which provide programmatic access to high |
| 4084 | 339 level function and class definitions provided by a module. The |
| 340 classes extract information from the parse tree and provide access to | |
| 341 the information at a useful semantic level, one function provides a | |
| 342 simple low-level pattern matching capability, and the other function | |
| 343 defines a high-level interface to the classes by handling file | |
| 344 operations on behalf of the caller. All source files mentioned here | |
| 345 which are not part of the Python installation are located in the | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
346 \file{Demo/parser/} directory of the distribution. |
| 4084 | 347 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
348 The dynamic nature of Python allows the programmer a great deal of |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
349 flexibility, but most modules need only a limited measure of this when |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
350 defining classes, functions, and methods. In this example, the only |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
351 definitions that will be considered are those which are defined in the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
352 top level of their context, e.g., a function defined by a \code{def} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
353 statement at column zero of a module, but not a function defined |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
354 within a branch of an \code{if} ... \code{else} construct, though |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
355 there are some good reasons for doing so in some situations. Nesting |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
356 of definitions will be handled by the code developed in the example. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
357 |
| 4084 | 358 To construct the upper-level extraction methods, we need to know what |
| 359 the parse tree structure looks like and how much of it we actually | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
360 need to be concerned about. Python uses a moderately deep parse tree |
| 4084 | 361 so there are a large number of intermediate nodes. It is important to |
| 362 read and understand the formal grammar used by Python. This is | |
| 363 specified in the file \file{Grammar/Grammar} in the distribution. | |
| 364 Consider the simplest case of interest when searching for docstrings: | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
365 a module consisting of a docstring and nothing else. (See file |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
366 \file{docstring.py}.) |
| 4084 | 367 |
| 5818 | 368 \bcode\begin{verbatim} |
| 4084 | 369 """Some documentation. |
| 370 """ | |
| 5818 | 371 \end{verbatim}\ecode |
| 372 % | |
| 4084 | 373 Using the interpreter to take a look at the parse tree, we find a |
| 374 bewildering mass of numbers and parentheses, with the documentation | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
375 buried deep in nested tuples. |
| 3198 | 376 |
| 5818 | 377 \bcode\begin{verbatim} |
| 4084 | 378 >>> import parser |
| 379 >>> import pprint | |
| 380 >>> ast = parser.suite(open('docstring.py').read()) | |
| 381 >>> tup = parser.ast2tuple(ast) | |
| 382 >>> pprint.pprint(tup) | |
| 383 (257, | |
| 384 (264, | |
| 385 (265, | |
| 386 (266, | |
| 387 (267, | |
| 388 (307, | |
| 389 (287, | |
| 390 (288, | |
| 391 (289, | |
| 392 (290, | |
| 393 (292, | |
| 394 (293, | |
| 395 (294, | |
| 396 (295, | |
| 397 (296, | |
| 398 (297, | |
| 399 (298, | |
| 400 (299, | |
| 401 (300, (3, '"""Some documentation.\012"""'))))))))))))))))), | |
| 402 (4, ''))), | |
| 403 (4, ''), | |
| 404 (0, '')) | |
| 5818 | 405 \end{verbatim}\ecode |
| 406 % | |
| 4084 | 407 The numbers at the first element of each node in the tree are the node |
| 408 types; they map directly to terminal and non-terminal symbols in the | |
| 409 grammar. Unfortunately, they are represented as integers in the | |
| 410 internal representation, and the Python structures generated do not | |
| 411 change that. However, the \code{symbol} and \code{token} modules | |
| 412 provide symbolic names for the node types and dictionaries which map | |
| 413 from the integers to the symbolic names for the node types. | |
| 3198 | 414 |
| 4084 | 415 In the output presented above, the outermost tuple contains four |
| 416 elements: the integer \code{257} and three additional tuples. Node | |
| 417 type \code{257} has the symbolic name \code{file_input}. Each of | |
| 418 these inner tuples contains an integer as the first element; these | |
| 419 integers, \code{264}, \code{4}, and \code{0}, represent the node types | |
| 420 \code{stmt}, \code{NEWLINE}, and \code{ENDMARKER}, respectively. | |
| 421 Note that these values may change depending on the version of Python | |
| 422 you are using; consult \file{symbol.py} and \file{token.py} for | |
| 423 details of the mapping. It should be fairly clear that the outermost | |
| 424 node is related primarily to the input source rather than the contents | |
| 425 of the file, and may be disregarded for the moment. The \code{stmt} | |
| 426 node is much more interesting. In particular, all docstrings are | |
| 427 found in subtrees which are formed exactly as this node is formed, | |
| 428 with the only difference being the string itself. The association | |
| 429 between the docstring in a similar tree and the defined entity (class, | |
| 430 function, or module) which it describes is given by the position of | |
| 431 the docstring subtree within the tree defining the described | |
| 432 structure. | |
| 3198 | 433 |
| 4084 | 434 By replacing the actual docstring with something to signify a variable |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
435 component of the tree, we allow a simple pattern matching approach to |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
436 check any given subtree for equivelence to the general pattern for |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
437 docstrings. Since the example demonstrates information extraction, we |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
438 can safely require that the tree be in tuple form rather than list |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
439 form, allowing a simple variable representation to be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
440 \code{['variable_name']}. A simple recursive function can implement |
| 4084 | 441 the pattern matching, returning a boolean and a dictionary of variable |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
442 name to value mappings. (See file \file{example.py}.) |
| 4084 | 443 |
| 5818 | 444 \bcode\begin{verbatim} |
| 4084 | 445 from types import ListType, TupleType |
| 446 | |
| 447 def match(pattern, data, vars=None): | |
| 448 if vars is None: | |
| 449 vars = {} | |
| 450 if type(pattern) is ListType: | |
| 451 vars[pattern[0]] = data | |
| 452 return 1, vars | |
| 453 if type(pattern) is not TupleType: | |
| 454 return (pattern == data), vars | |
| 455 if len(data) != len(pattern): | |
| 456 return 0, vars | |
| 457 for pattern, data in map(None, pattern, data): | |
| 458 same, vars = match(pattern, data, vars) | |
| 459 if not same: | |
| 460 break | |
| 461 return same, vars | |
| 5818 | 462 \end{verbatim}\ecode |
| 463 % | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
464 Using this simple representation for syntactic variables and the symbolic |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
465 node types, the pattern for the candidate docstring subtrees becomes |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
466 fairly readable. (See file \file{example.py}.) |
| 3198 | 467 |
| 5818 | 468 \bcode\begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
469 import symbol |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
470 import token |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
471 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
472 DOCSTRING_STMT_PATTERN = ( |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
473 symbol.stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
474 (symbol.simple_stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
475 (symbol.small_stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
476 (symbol.expr_stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
477 (symbol.testlist, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
478 (symbol.test, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
479 (symbol.and_test, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
480 (symbol.not_test, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
481 (symbol.comparison, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
482 (symbol.expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
483 (symbol.xor_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
484 (symbol.and_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
485 (symbol.shift_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
486 (symbol.arith_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
487 (symbol.term, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
488 (symbol.factor, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
489 (symbol.power, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
490 (symbol.atom, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
491 (token.STRING, ['docstring']) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
492 )))))))))))))))), |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
493 (token.NEWLINE, '') |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
494 )) |
| 5818 | 495 \end{verbatim}\ecode |
| 496 % | |
| 4084 | 497 Using the \code{match()} function with this pattern, extracting the |
| 498 module docstring from the parse tree created previously is easy: | |
| 499 | |
| 5818 | 500 \bcode\begin{verbatim} |
| 4084 | 501 >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1]) |
| 502 >>> found | |
| 503 1 | |
| 504 >>> vars | |
| 505 {'docstring': '"""Some documentation.\012"""'} | |
| 5818 | 506 \end{verbatim}\ecode |
| 507 % | |
| 4084 | 508 Once specific data can be extracted from a location where it is |
| 509 expected, the question of where information can be expected | |
| 510 needs to be answered. When dealing with docstrings, the answer is | |
| 511 fairly simple: the docstring is the first \code{stmt} node in a code | |
| 512 block (\code{file_input} or \code{suite} node types). A module | |
| 513 consists of a single \code{file_input} node, and class and function | |
| 514 definitions each contain exactly one \code{suite} node. Classes and | |
| 515 functions are readily identified as subtrees of code block nodes which | |
| 516 start with \code{(stmt, (compound_stmt, (classdef, ...} or | |
| 517 \code{(stmt, (compound_stmt, (funcdef, ...}. Note that these subtrees | |
| 518 cannot be matched by \code{match()} since it does not support multiple | |
| 519 sibling nodes to match without regard to number. A more elaborate | |
| 520 matching function could be used to overcome this limitation, but this | |
| 521 is sufficient for the example. | |
| 522 | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
523 Given the ability to determine whether a statement might be a |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
524 docstring and extract the actual string from the statement, some work |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
525 needs to be performed to walk the parse tree for an entire module and |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
526 extract information about the names defined in each context of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
527 module and associate any docstrings with the names. The code to |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
528 perform this work is not complicated, but bears some explanation. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
529 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
530 The public interface to the classes is straightforward and should |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
531 probably be somewhat more flexible. Each ``major'' block of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
532 module is described by an object providing several methods for inquiry |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
533 and a constructor which accepts at least the subtree of the complete |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
534 parse tree which it represents. The \code{ModuleInfo} constructor |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
535 accepts an optional \code{\var{name}} parameter since it cannot |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
536 otherwise determine the name of the module. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
537 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
538 The public classes include \code{ClassInfo}, \code{FunctionInfo}, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
539 and \code{ModuleInfo}. All objects provide the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
540 methods \code{get_name()}, \code{get_docstring()}, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
541 \code{get_class_names()}, and \code{get_class_info()}. The |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
542 \code{ClassInfo} objects support \code{get_method_names()} and |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
543 \code{get_method_info()} while the other classes provide |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
544 \code{get_function_names()} and \code{get_function_info()}. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
545 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
546 Within each of the forms of code block that the public classes |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
547 represent, most of the required information is in the same form and is |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
548 accessed in the same way, with classes having the distinction that |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
549 functions defined at the top level are referred to as ``methods.'' |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
550 Since the difference in nomenclature reflects a real semantic |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
551 distinction from functions defined outside of a class, the |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
552 implementation needs to maintain the distinction. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
553 Hence, most of the functionality of the public classes can be |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
554 implemented in a common base class, \code{SuiteInfoBase}, with the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
555 accessors for function and method information provided elsewhere. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
556 Note that there is only one class which represents function and method |
|
5032
be31e7e19241
(libparser.tex): Fix typo found by Mark Hammond.
Fred Drake <fdrake@acm.org>
parents:
4728
diff
changeset
|
557 information; this parallels the use of the \code{def} statement to |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
558 define both types of elements. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
559 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
560 Most of the accessor functions are declared in \code{SuiteInfoBase} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
561 and do not need to be overriden by subclasses. More importantly, the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
562 extraction of most information from a parse tree is handled through a |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
563 method called by the \code{SuiteInfoBase} constructor. The example |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
564 code for most of the classes is clear when read alongside the formal |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
565 grammar, but the method which recursively creates new information |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
566 objects requires further examination. Here is the relevant part of |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
567 the \code{SuiteInfoBase} definition from \file{example.py}: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
568 |
| 5818 | 569 \bcode\begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
570 class SuiteInfoBase: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
571 _docstring = '' |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
572 _name = '' |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
573 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
574 def __init__(self, tree = None): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
575 self._class_info = {} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
576 self._function_info = {} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
577 if tree: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
578 self._extract_info(tree) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
579 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
580 def _extract_info(self, tree): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
581 # extract docstring |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
582 if len(tree) == 2: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
583 found, vars = match(DOCSTRING_STMT_PATTERN[1], tree[1]) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
584 else: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
585 found, vars = match(DOCSTRING_STMT_PATTERN, tree[3]) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
586 if found: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
587 self._docstring = eval(vars['docstring']) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
588 # discover inner definitions |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
589 for node in tree[1:]: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
590 found, vars = match(COMPOUND_STMT_PATTERN, node) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
591 if found: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
592 cstmt = vars['compound'] |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
593 if cstmt[0] == symbol.funcdef: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
594 name = cstmt[2][1] |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
595 self._function_info[name] = FunctionInfo(cstmt) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
596 elif cstmt[0] == symbol.classdef: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
597 name = cstmt[2][1] |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
598 self._class_info[name] = ClassInfo(cstmt) |
| 5818 | 599 \end{verbatim}\ecode |
| 600 % | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
601 After initializing some internal state, the constructor calls the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
602 \code{_extract_info()} method. This method performs the bulk of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
603 information extraction which takes place in the entire example. The |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
604 extraction has two distinct phases: the location of the docstring for |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
605 the parse tree passed in, and the discovery of additional definitions |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
606 within the code block represented by the parse tree. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
607 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
608 The initial \code{if} test determines whether the nested suite is of |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
609 the ``short form'' or the ``long form.'' The short form is used when |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
610 the code block is on the same line as the definition of the code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
611 block, as in |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
612 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
613 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
614 def square(x): "Square an argument."; return x ** 2 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
615 \end{verbatim} |
| 5818 | 616 % |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
617 while the long form uses an indented block and allows nested |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
618 definitions: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
619 |
| 5818 | 620 \bcode\begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
621 def make_power(exp): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
622 "Make a function that raises an argument to the exponent `exp'." |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
623 def raiser(x, y=exp): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
624 return x ** y |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
625 return raiser |
| 5818 | 626 \end{verbatim}\ecode |
| 627 % | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
628 When the short form is used, the code block may contain a docstring as |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
629 the first, and possibly only, \code{small_stmt} element. The |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
630 extraction of such a docstring is slightly different and requires only |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
631 a portion of the complete pattern used in the more common case. As |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
632 implemented, the docstring will only be found if there is only |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
633 one \code{small_stmt} node in the \code{simple_stmt} node. Since most |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
634 functions and methods which use the short form do not provide a |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
635 docstring, this may be considered sufficient. The extraction of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
636 docstring proceeds using the \code{match()} function as described |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
637 above, and the value of the docstring is stored as an attribute of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
638 \code{SuiteInfoBase} object. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
639 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
640 After docstring extraction, a simple definition discovery |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
641 algorithm operates on the \code{stmt} nodes of the \code{suite} node. The |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
642 special case of the short form is not tested; since there are no |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
643 \code{stmt} nodes in the short form, the algorithm will silently skip |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
644 the single \code{simple_stmt} node and correctly not discover any |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
645 nested definitions. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
646 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
647 Each statement in the code block is categorized as |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
648 a class definition, function or method definition, or |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
649 something else. For the definition statements, the name of the |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
650 element defined is extracted and a representation object |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
651 appropriate to the definition is created with the defining subtree |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
652 passed as an argument to the constructor. The repesentation objects |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
653 are stored in instance variables and may be retrieved by name using |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
654 the appropriate accessor methods. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
655 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
656 The public classes provide any accessors required which are more |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
657 specific than those provided by the \code{SuiteInfoBase} class, but |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
658 the real extraction algorithm remains common to all forms of code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
659 blocks. A high-level function can be used to extract the complete set |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
660 of information from a source file. (See file \file{example.py}.) |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
661 |
| 5818 | 662 \bcode\begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
663 def get_docs(fileName): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
664 source = open(fileName).read() |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
665 import os |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
666 basename = os.path.basename(os.path.splitext(fileName)[0]) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
667 import parser |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
668 ast = parser.suite(source) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
669 tup = parser.ast2tuple(ast) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
670 return ModuleInfo(tup, basename) |
| 5818 | 671 \end{verbatim}\ecode |
| 672 % | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
673 This provides an easy-to-use interface to the documentation of a |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
674 module. If information is required which is not extracted by the code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
675 of this example, the code may be extended at clearly defined points to |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
676 provide additional capabilities. |
| 4084 | 677 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
678 \begin{seealso} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
679 |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
680 \seemodule{symbol}% |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
681 {useful constants representing internal nodes of the parse tree} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
682 |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
683 \seemodule{token}% |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
684 {useful constants representing leaf nodes of the parse tree and |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
685 functions for testing node values} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
686 |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
687 \end{seealso} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
688 |
| 4084 | 689 |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
690 \section{Standard Module \sectcode{symbol}} |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
691 \label{module-symbol} |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
692 \stmodindex{symbol} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
693 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
694 This module provides constants which represent the numeric values of |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
695 internal nodes of the parse tree. Unlike most Python constants, these |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
696 use lower-case names. Refer to the file \file{Grammar/Grammar} in the |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
697 Python distribution for the defintions of the names in the context of |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
698 the language grammar. The specific numeric values which the names map |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
699 to may change between Python versions. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
700 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
701 This module also provides one additional data object: |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
702 |
|
6648
0f2498b847bf
Removed leading comment about this being "introductory" documentation;
Fred Drake <fdrake@acm.org>
parents:
6408
diff
changeset
|
703 \renewcommand{\indexsubitem}{(in module symbol)} |
|
0f2498b847bf
Removed leading comment about this being "introductory" documentation;
Fred Drake <fdrake@acm.org>
parents:
6408
diff
changeset
|
704 |
|
0f2498b847bf
Removed leading comment about this being "introductory" documentation;
Fred Drake <fdrake@acm.org>
parents:
6408
diff
changeset
|
705 |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
706 \begin{datadesc}{sym_name} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
707 Dictionary mapping the numeric values of the constants defined in this |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
708 module back to name strings, allowing more human-readable |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
709 representation of parse trees to be generated. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
710 \end{datadesc} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
711 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
712 \begin{seealso} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
713 \seemodule{parser}{second example uses this module} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
714 \end{seealso} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
715 |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
716 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
717 \section{Standard Module \sectcode{token}} |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
718 \label{module-token} |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
719 \stmodindex{token} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
720 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
721 This module provides constants which represent the numeric values of |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
722 leaf nodes of the parse tree (terminal tokens). Refer to the file |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
723 \file{Grammar/Grammar} in the Python distribution for the defintions |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
724 of the names in the context of the language grammar. The specific |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
725 numeric values which the names map to may change between Python |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
726 versions. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
727 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
728 This module also provides one data object and some functions. The |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
729 functions mirror definitions in the Python C header files. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
730 |
|
6648
0f2498b847bf
Removed leading comment about this being "introductory" documentation;
Fred Drake <fdrake@acm.org>
parents:
6408
diff
changeset
|
731 \renewcommand{\indexsubitem}{(in module token)} |
|
0f2498b847bf
Removed leading comment about this being "introductory" documentation;
Fred Drake <fdrake@acm.org>
parents:
6408
diff
changeset
|
732 |
|
0f2498b847bf
Removed leading comment about this being "introductory" documentation;
Fred Drake <fdrake@acm.org>
parents:
6408
diff
changeset
|
733 |
|
6408
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
734 \begin{datadesc}{tok_name} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
735 Dictionary mapping the numeric values of the constants defined in this |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
736 module back to name strings, allowing more human-readable |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
737 representation of parse trees to be generated. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
738 \end{datadesc} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
739 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
740 \begin{funcdesc}{ISTERMINAL}{x} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
741 Return true for terminal token values. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
742 \end{funcdesc} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
743 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
744 \begin{funcdesc}{ISNONTERMINAL}{x} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
745 Return true for non-terminal token values. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
746 \end{funcdesc} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
747 |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
748 \begin{funcdesc}{ISEOF}{x} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
749 Return true if \var{x} is the marker indicating the end of input. |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
750 \end{funcdesc} |
|
184093d96d2a
Minor nits (Lib/token.py is a file, token is a module...).
Fred Drake <fdrake@acm.org>
parents:
5818
diff
changeset
|
751 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
752 \begin{seealso} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
753 \seemodule{parser}{second example uses this module} |
|
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
754 \end{seealso} |