Mercurial > cpython-lru_cache-weakref
annotate Doc/lib/libparser.tex @ 13625:fdfd1049f36e legacy-trunk
Rob W. W. Hooft's spelling fixes for the Library Reference. I hope
SourceForge doesn't choke on this batch :-)
I'm not entirely sure this is 100% correct. The patch changes an
\index{persistency} to \index{presistence}, and I don't know what \index{}
does. But it seems to do so persi--er, consistently, so I hope it isn't a
problem.
| author | Thomas Wouters <thomas@python.org> |
|---|---|
| date | Sun, 16 Jul 2000 19:01:10 +0000 |
| parents | 3f06f4765ba4 |
| children | c7c404b35f33 |
| rev | line source |
|---|---|
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
1 \section{\module{parser} --- |
|
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
2 Access Python parse trees} |
|
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
3 |
| 3198 | 4 % Copyright 1995 Virginia Polytechnic Institute and State University |
| 5 % and Fred L. Drake, Jr. This copyright notice must be distributed on | |
| 6 % all copies, but this document otherwise may be distributed as part | |
| 7 % of the Python distribution. No fee may be charged for this document | |
| 8 % in any representation, either on paper or electronically. This | |
| 9 % restriction does not affect other elements in a distributed package | |
| 10 % in any way. | |
|
10245
bdbfe0ec970f
Use more current approach to recording authors.
Fred Drake <fdrake@acm.org>
parents:
8918
diff
changeset
|
11 |
|
8716
760dcf197048
Adjusted to use the new module synopsis support macros.
Fred Drake <fdrake@acm.org>
parents:
8040
diff
changeset
|
12 \declaremodule{builtin}{parser} |
|
10245
bdbfe0ec970f
Use more current approach to recording authors.
Fred Drake <fdrake@acm.org>
parents:
8918
diff
changeset
|
13 \modulesynopsis{Access parse trees for Python source code.} |
| 8918 | 14 \moduleauthor{Fred L. Drake, Jr.}{fdrake@acm.org} |
| 15 \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org} | |
|
8716
760dcf197048
Adjusted to use the new module synopsis support macros.
Fred Drake <fdrake@acm.org>
parents:
8040
diff
changeset
|
16 |
|
760dcf197048
Adjusted to use the new module synopsis support macros.
Fred Drake <fdrake@acm.org>
parents:
8040
diff
changeset
|
17 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
18 \index{parsing!Python source code} |
| 3198 | 19 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
20 The \module{parser} module provides an interface to Python's internal |
| 3198 | 21 parser and byte-code compiler. The primary purpose for this interface |
| 22 is to allow Python code to edit the parse tree of a Python expression | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
23 and create executable code from this. This is better than trying |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
24 to parse and modify an arbitrary Python code fragment as a string |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
25 because parsing is performed in a manner identical to the code |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
26 forming the application. It is also faster. |
| 3198 | 27 |
| 28 There are a few things to note about this module which are important | |
| 29 to making use of the data structures created. This is not a tutorial | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
30 on editing the parse trees for Python code, but some examples of using |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
31 the \module{parser} module are presented. |
| 3198 | 32 |
| 33 Most importantly, a good understanding of the Python grammar processed | |
| 34 by the internal parser is required. For full information on the | |
|
11463
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
35 language syntax, refer to the \citetitle[../ref/ref.html]{Python |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
36 Language Reference}. The parser itself is created from a grammar |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
37 specification defined in the file \file{Grammar/Grammar} in the |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
38 standard Python distribution. The parse trees stored in the AST |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
39 objects created by this module are the actual output from the internal |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
40 parser when created by the \function{expr()} or \function{suite()} |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
41 functions, described below. The AST objects created by |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
42 \function{sequence2ast()} faithfully simulate those structures. Be |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
43 aware that the values of the sequences which are considered |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
44 ``correct'' will vary from one version of Python to another as the |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
45 formal grammar for the language is revised. However, transporting |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
46 code from one Python version to another as source text will always |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
47 allow correct parse trees to be created in the target version, with |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
48 the only restriction being that migrating to an older version of the |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
49 interpreter will not support more recent language constructs. The |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
50 parse trees are not typically compatible from one version to another, |
|
17e12305973e
Use \citetitle and \programopt as appropriate.
Fred Drake <fdrake@acm.org>
parents:
11265
diff
changeset
|
51 whereas source code has always been forward-compatible. |
| 3198 | 52 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
53 Each element of the sequences returned by \function{ast2list()} or |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
54 \function{ast2tuple()} has a simple form. Sequences representing |
| 4084 | 55 non-terminal elements in the grammar always have a length greater than |
| 56 one. The first element is an integer which identifies a production in | |
| 57 the grammar. These integers are given symbolic names in the C header | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
58 file \file{Include/graminit.h} and the Python module |
|
10703
e4e800a78d8d
Make internal module references hyperlinks wherever it makes sense.
Fred Drake <fdrake@acm.org>
parents:
10245
diff
changeset
|
59 \refmodule{symbol}. Each additional element of the sequence represents |
| 4084 | 60 a component of the production as recognized in the input string: these |
| 61 are always sequences which have the same form as the parent. An | |
| 62 important aspect of this structure which should be noted is that | |
| 63 keywords used to identify the parent node type, such as the keyword | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
64 \keyword{if} in an \constant{if_stmt}, are included in the node tree without |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
65 any special treatment. For example, the \keyword{if} keyword is |
| 3198 | 66 represented by the tuple \code{(1, 'if')}, where \code{1} is the |
| 11160 | 67 numeric value associated with all \constant{NAME} tokens, including |
| 4084 | 68 variable and function names defined by the user. In an alternate form |
| 69 returned when line number information is requested, the same token | |
| 70 might be represented as \code{(1, 'if', 12)}, where the \code{12} | |
| 71 represents the line number at which the terminal symbol was found. | |
| 3198 | 72 |
| 73 Terminal elements are represented in much the same way, but without | |
| 74 any child elements and the addition of the source text which was | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
75 identified. The example of the \keyword{if} keyword above is |
| 3198 | 76 representative. The various types of terminal symbols are defined in |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
77 the C header file \file{Include/token.h} and the Python module |
|
10703
e4e800a78d8d
Make internal module references hyperlinks wherever it makes sense.
Fred Drake <fdrake@acm.org>
parents:
10245
diff
changeset
|
78 \refmodule{token}. |
| 3198 | 79 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
80 The AST objects are not required to support the functionality of this |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
81 module, but are provided for three purposes: to allow an application |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
82 to amortize the cost of processing complex parse trees, to provide a |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
83 parse tree representation which conserves memory space when compared |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
84 to the Python list or tuple representation, and to ease the creation |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
85 of additional modules in C which manipulate parse trees. A simple |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
86 ``wrapper'' class may be created in Python to hide the use of AST |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
87 objects. |
| 3198 | 88 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
89 The \module{parser} module defines functions for a few distinct |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
90 purposes. The most important purposes are to create AST objects and |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
91 to convert AST objects to other representations such as parse trees |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
92 and compiled code objects, but there are also functions which serve to |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
93 query the type of parse tree represented by an AST object. |
| 3198 | 94 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
95 |
|
10845
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
96 \begin{seealso} |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
97 \seemodule{symbol}{Useful constants representing internal nodes of |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
98 the parse tree.} |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
99 \seemodule{token}{Useful constants representing leaf nodes of the |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
100 parse tree and functions for testing node values.} |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
101 \end{seealso} |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
102 |
|
5ef4b3ca5564
Move the "See also:" section so it appears are part of the top HTML
Fred Drake <fdrake@acm.org>
parents:
10703
diff
changeset
|
103 |
| 11160 | 104 \subsection{Creating AST Objects \label{Creating ASTs}} |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
105 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
106 AST objects may be created from source code or from a parse tree. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
107 When creating an AST object from source, different functions are used |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
108 to create the \code{'eval'} and \code{'exec'} forms. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
109 |
|
11265
8336188a3c10
Name the parameter to expr() and suite() "source" instead of "string".
Fred Drake <fdrake@acm.org>
parents:
11160
diff
changeset
|
110 \begin{funcdesc}{expr}{source} |
|
8336188a3c10
Name the parameter to expr() and suite() "source" instead of "string".
Fred Drake <fdrake@acm.org>
parents:
11160
diff
changeset
|
111 The \function{expr()} function parses the parameter \var{source} |
|
12640
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
112 as if it were an input to \samp{compile(\var{source}, 'file.py', |
|
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
113 'eval')}. If the parse succeeds, an AST object is created to hold the |
|
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
114 internal parse tree representation, otherwise an appropriate exception |
|
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
115 is thrown. |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
116 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
117 |
|
11265
8336188a3c10
Name the parameter to expr() and suite() "source" instead of "string".
Fred Drake <fdrake@acm.org>
parents:
11160
diff
changeset
|
118 \begin{funcdesc}{suite}{source} |
|
8336188a3c10
Name the parameter to expr() and suite() "source" instead of "string".
Fred Drake <fdrake@acm.org>
parents:
11160
diff
changeset
|
119 The \function{suite()} function parses the parameter \var{source} |
|
12640
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
120 as if it were an input to \samp{compile(\var{source}, 'file.py', |
|
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
121 'exec')}. If the parse succeeds, an AST object is created to hold the |
|
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
122 internal parse tree representation, otherwise an appropriate exception |
|
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
123 is thrown. |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
124 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
125 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
126 \begin{funcdesc}{sequence2ast}{sequence} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
127 This function accepts a parse tree represented as a sequence and |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
128 builds an internal representation if possible. If it can validate |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
129 that the tree conforms to the Python grammar and all nodes are valid |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
130 node types in the host version of Python, an AST object is created |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
131 from the internal representation and returned to the called. If there |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
132 is a problem creating the internal representation, or if the tree |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
133 cannot be validated, a \exception{ParserError} exception is thrown. An AST |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
134 object created this way should not be assumed to compile correctly; |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
135 normal exceptions thrown by compilation may still be initiated when |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
136 the AST object is passed to \function{compileast()}. This may indicate |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
137 problems not related to syntax (such as a \exception{MemoryError} |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
138 exception), but may also be due to constructs such as the result of |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
139 parsing \code{del f(0)}, which escapes the Python parser but is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
140 checked by the bytecode compiler. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
141 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
142 Sequences representing terminal tokens may be represented as either |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
143 two-element lists of the form \code{(1, 'name')} or as three-element |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
144 lists of the form \code{(1, 'name', 56)}. If the third element is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
145 present, it is assumed to be a valid line number. The line number |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
146 may be specified for any subset of the terminal symbols in the input |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
147 tree. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
148 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
149 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
150 \begin{funcdesc}{tuple2ast}{sequence} |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
151 This is the same function as \function{sequence2ast()}. This entry point |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
152 is maintained for backward compatibility. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
153 \end{funcdesc} |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
154 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
155 |
| 11160 | 156 \subsection{Converting AST Objects \label{Converting ASTs}} |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
157 |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
158 AST objects, regardless of the input used to create them, may be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
159 converted to parse trees represented as list- or tuple- trees, or may |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
160 be compiled into executable code objects. Parse trees may be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
161 extracted with or without line numbering information. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
162 |
| 7809 | 163 \begin{funcdesc}{ast2list}{ast\optional{, line_info}} |
| 4084 | 164 This function accepts an AST object from the caller in |
| 11160 | 165 \var{ast} and returns a Python list representing the |
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
166 equivalent parse tree. The resulting list representation can be used |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
167 for inspection or the creation of a new parse tree in list form. This |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
168 function does not fail so long as memory is available to build the |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
169 list representation. If the parse tree will only be used for |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
170 inspection, \function{ast2tuple()} should be used instead to reduce memory |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
171 consumption and fragmentation. When the list representation is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
172 required, this function is significantly faster than retrieving a |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
173 tuple representation and converting that to nested lists. |
| 4084 | 174 |
| 11160 | 175 If \var{line_info} is true, line number information will be |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
176 included for all terminal tokens as a third element of the list |
|
4591
8cdd94a3d02d
(libparser.tex): Added note that the line number for a terminal token
Fred Drake <fdrake@acm.org>
parents:
4286
diff
changeset
|
177 representing the token. Note that the line number provided specifies |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
178 the line on which the token \emph{ends}. This information is |
|
4591
8cdd94a3d02d
(libparser.tex): Added note that the line number for a terminal token
Fred Drake <fdrake@acm.org>
parents:
4286
diff
changeset
|
179 omitted if the flag is false or omitted. |
| 4084 | 180 \end{funcdesc} |
| 181 | |
| 7809 | 182 \begin{funcdesc}{ast2tuple}{ast\optional{, line_info}} |
| 3198 | 183 This function accepts an AST object from the caller in |
| 11160 | 184 \var{ast} and returns a Python tuple representing the |
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
185 equivalent parse tree. Other than returning a tuple instead of a |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
186 list, this function is identical to \function{ast2list()}. |
| 4084 | 187 |
| 11160 | 188 If \var{line_info} is true, line number information will be |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
189 included for all terminal tokens as a third element of the list |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
190 representing the token. This information is omitted if the flag is |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
191 false or omitted. |
| 3198 | 192 \end{funcdesc} |
| 193 | |
|
7712
0cbc9df882b6
Change "\," to just "," in function signatures. This is easier to maintain,
Fred Drake <fdrake@acm.org>
parents:
7607
diff
changeset
|
194 \begin{funcdesc}{compileast}{ast\optional{, filename\code{ = '<ast>'}}} |
| 3198 | 195 The Python byte compiler can be invoked on an AST object to produce |
| 11160 | 196 code objects which can be used as part of an \keyword{exec} statement or |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
197 a call to the built-in \function{eval()}\bifuncindex{eval} function. |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
198 This function provides the interface to the compiler, passing the |
| 11160 | 199 internal parse tree from \var{ast} to the parser, using the |
| 200 source file name specified by the \var{filename} parameter. | |
| 201 The default value supplied for \var{filename} indicates that | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
202 the source was an AST object. |
| 4084 | 203 |
| 204 Compiling an AST object may result in exceptions related to | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
205 compilation; an example would be a \exception{SyntaxError} caused by the |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
206 parse tree for \code{del f(0)}: this statement is considered legal |
| 4084 | 207 within the formal grammar for Python but is not a legal language |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
208 construct. The \exception{SyntaxError} raised for this condition is |
| 4084 | 209 actually generated by the Python byte-compiler normally, which is why |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
210 it can be raised at this point by the \module{parser} module. Most |
| 4084 | 211 causes of compilation failure can be diagnosed programmatically by |
| 212 inspection of the parse tree. | |
| 3198 | 213 \end{funcdesc} |
| 214 | |
| 215 | |
| 11160 | 216 \subsection{Queries on AST Objects \label{Querying ASTs}} |
| 3198 | 217 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
218 Two functions are provided which allow an application to determine if |
| 7809 | 219 an AST was created as an expression or a suite. Neither of these |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
220 functions can be used to determine if an AST was created from source |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
221 code via \function{expr()} or \function{suite()} or from a parse tree |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
222 via \function{sequence2ast()}. |
| 3198 | 223 |
| 224 \begin{funcdesc}{isexpr}{ast} | |
| 11160 | 225 When \var{ast} represents an \code{'eval'} form, this function |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
226 returns true, otherwise it returns false. This is useful, since code |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
227 objects normally cannot be queried for this information using existing |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
228 built-in functions. Note that the code objects created by |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
229 \function{compileast()} cannot be queried like this either, and are |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
230 identical to those created by the built-in |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
231 \function{compile()}\bifuncindex{compile} function. |
| 3198 | 232 \end{funcdesc} |
| 233 | |
| 234 | |
| 235 \begin{funcdesc}{issuite}{ast} | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
236 This function mirrors \function{isexpr()} in that it reports whether an |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
237 AST object represents an \code{'exec'} form, commonly known as a |
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
238 ``suite.'' It is not safe to assume that this function is equivalent |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
239 to \samp{not isexpr(\var{ast})}, as additional syntactic fragments may |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
240 be supported in the future. |
| 3198 | 241 \end{funcdesc} |
| 242 | |
| 243 | |
| 11160 | 244 \subsection{Exceptions and Error Handling \label{AST Errors}} |
| 3198 | 245 |
| 246 The parser module defines a single exception, but may also pass other | |
| 247 built-in exceptions from other portions of the Python runtime | |
| 248 environment. See each function for information about the exceptions | |
| 249 it can raise. | |
| 250 | |
| 251 \begin{excdesc}{ParserError} | |
| 252 Exception raised when a failure occurs within the parser module. This | |
| 253 is generally produced for validation failures rather than the built in | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
254 \exception{SyntaxError} thrown during normal parsing. |
| 3198 | 255 The exception argument is either a string describing the reason of the |
| 4084 | 256 failure or a tuple containing a sequence causing the failure from a parse |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
257 tree passed to \function{sequence2ast()} and an explanatory string. Calls to |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
258 \function{sequence2ast()} need to be able to handle either type of exception, |
| 3198 | 259 while calls to other functions in the module will only need to be |
| 260 aware of the simple string values. | |
| 261 \end{excdesc} | |
| 262 | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
263 Note that the functions \function{compileast()}, \function{expr()}, and |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
264 \function{suite()} may throw exceptions which are normally thrown by the |
| 3198 | 265 parsing and compilation process. These include the built in |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
266 exceptions \exception{MemoryError}, \exception{OverflowError}, |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
267 \exception{SyntaxError}, and \exception{SystemError}. In these cases, these |
| 3198 | 268 exceptions carry all the meaning normally associated with them. Refer |
| 269 to the descriptions of each function for detailed information. | |
| 270 | |
| 271 | |
| 11160 | 272 \subsection{AST Objects \label{AST Objects}} |
| 4084 | 273 |
|
7879
427a9b1c9abd
Move a short paragraph outside the description of ASTType.
Fred Drake <fdrake@acm.org>
parents:
7877
diff
changeset
|
274 Ordered and equality comparisons are supported between AST objects. |
|
10703
e4e800a78d8d
Make internal module references hyperlinks wherever it makes sense.
Fred Drake <fdrake@acm.org>
parents:
10245
diff
changeset
|
275 Pickling of AST objects (using the \refmodule{pickle} module) is also |
|
8035
6c594988572a
Add a note that pickling is supported.
Fred Drake <fdrake@acm.org>
parents:
7933
diff
changeset
|
276 supported. |
|
7879
427a9b1c9abd
Move a short paragraph outside the description of ASTType.
Fred Drake <fdrake@acm.org>
parents:
7877
diff
changeset
|
277 |
| 7607 | 278 \begin{datadesc}{ASTType} |
| 279 The type of the objects returned by \function{expr()}, | |
| 280 \function{suite()} and \function{sequence2ast()}. | |
| 281 \end{datadesc} | |
| 4084 | 282 |
| 283 | |
|
8040
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
284 AST objects have the following methods: |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
285 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
286 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
287 \begin{methoddesc}[AST]{compile}{\optional{filename}} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
288 Same as \code{compileast(\var{ast}, \var{filename})}. |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
289 \end{methoddesc} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
290 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
291 \begin{methoddesc}[AST]{isexpr}{} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
292 Same as \code{isexpr(\var{ast})}. |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
293 \end{methoddesc} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
294 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
295 \begin{methoddesc}[AST]{issuite}{} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
296 Same as \code{issuite(\var{ast})}. |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
297 \end{methoddesc} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
298 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
299 \begin{methoddesc}[AST]{tolist}{\optional{line_info}} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
300 Same as \code{ast2list(\var{ast}, \var{line_info})}. |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
301 \end{methoddesc} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
302 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
303 \begin{methoddesc}[AST]{totuple}{\optional{line_info}} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
304 Same as \code{ast2tuple(\var{ast}, \var{line_info})}. |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
305 \end{methoddesc} |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
306 |
|
df0fa5dc599a
No more threats, just list the new methods under "AST Objects".
Fred Drake <fdrake@acm.org>
parents:
8035
diff
changeset
|
307 |
| 11160 | 308 \subsection{Examples \label{AST Examples}} |
| 3198 | 309 |
| 4084 | 310 The parser modules allows operations to be performed on the parse tree |
| 311 of Python source code before the bytecode is generated, and provides | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
312 for inspection of the parse tree for information gathering purposes. |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
313 Two examples are presented. The simple example demonstrates emulation |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
314 of the \function{compile()}\bifuncindex{compile} built-in function and |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
315 the complex example shows the use of a parse tree for information |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
316 discovery. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
317 |
|
7879
427a9b1c9abd
Move a short paragraph outside the description of ASTType.
Fred Drake <fdrake@acm.org>
parents:
7877
diff
changeset
|
318 \subsubsection{Emulation of \function{compile()}} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
319 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
320 While many useful operations may take place between parsing and |
| 4084 | 321 bytecode generation, the simplest operation is to do nothing. For |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
322 this purpose, using the \module{parser} module to produce an |
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
323 intermediate data structure is equivalent to the code |
| 4084 | 324 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
325 \begin{verbatim} |
|
12640
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
326 >>> code = compile('a + 5', 'file.py', 'eval') |
| 4084 | 327 >>> a = 5 |
| 328 >>> eval(code) | |
| 329 10 | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
330 \end{verbatim} |
| 7809 | 331 |
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
332 The equivalent operation using the \module{parser} module is somewhat |
| 4084 | 333 longer, and allows the intermediate internal parse tree to be retained |
| 334 as an AST object: | |
| 3198 | 335 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
336 \begin{verbatim} |
| 3198 | 337 >>> import parser |
| 338 >>> ast = parser.expr('a + 5') | |
|
12640
3f06f4765ba4
Fix references to the built-in compile() that don't include the
Fred Drake <fdrake@acm.org>
parents:
12223
diff
changeset
|
339 >>> code = ast.compile('file.py') |
| 3198 | 340 >>> a = 5 |
| 341 >>> eval(code) | |
| 342 10 | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
343 \end{verbatim} |
| 7809 | 344 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
345 An application which needs both AST and code objects can package this |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
346 code into readily available functions: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
347 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
348 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
349 import parser |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
350 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
351 def load_suite(source_string): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
352 ast = parser.suite(source_string) |
| 11160 | 353 return ast, ast.compile() |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
354 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
355 def load_expression(source_string): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
356 ast = parser.expr(source_string) |
| 11160 | 357 return ast, ast.compile() |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
358 \end{verbatim} |
| 7809 | 359 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
360 \subsubsection{Information Discovery} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
361 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
362 Some applications benefit from direct access to the parse tree. The |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
363 remainder of this section demonstrates how the parse tree provides |
| 11160 | 364 access to module documentation defined in |
| 365 docstrings\index{string!documentation}\index{docstrings} without | |
| 366 requiring that the code being examined be loaded into a running | |
| 367 interpreter via \keyword{import}. This can be very useful for | |
| 368 performing analyses of untrusted code. | |
| 3198 | 369 |
| 4084 | 370 Generally, the example will demonstrate how the parse tree may be |
| 371 traversed to distill interesting information. Two functions and a set | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
372 of classes are developed which provide programmatic access to high |
| 4084 | 373 level function and class definitions provided by a module. The |
| 374 classes extract information from the parse tree and provide access to | |
| 375 the information at a useful semantic level, one function provides a | |
| 376 simple low-level pattern matching capability, and the other function | |
| 377 defines a high-level interface to the classes by handling file | |
| 378 operations on behalf of the caller. All source files mentioned here | |
| 379 which are not part of the Python installation are located in the | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
380 \file{Demo/parser/} directory of the distribution. |
| 4084 | 381 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
382 The dynamic nature of Python allows the programmer a great deal of |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
383 flexibility, but most modules need only a limited measure of this when |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
384 defining classes, functions, and methods. In this example, the only |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
385 definitions that will be considered are those which are defined in the |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
386 top level of their context, e.g., a function defined by a \keyword{def} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
387 statement at column zero of a module, but not a function defined |
| 11160 | 388 within a branch of an \keyword{if} ... \keyword{else} construct, though |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
389 there are some good reasons for doing so in some situations. Nesting |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
390 of definitions will be handled by the code developed in the example. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
391 |
| 4084 | 392 To construct the upper-level extraction methods, we need to know what |
| 393 the parse tree structure looks like and how much of it we actually | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
394 need to be concerned about. Python uses a moderately deep parse tree |
| 4084 | 395 so there are a large number of intermediate nodes. It is important to |
| 396 read and understand the formal grammar used by Python. This is | |
| 397 specified in the file \file{Grammar/Grammar} in the distribution. | |
| 398 Consider the simplest case of interest when searching for docstrings: | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
399 a module consisting of a docstring and nothing else. (See file |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
400 \file{docstring.py}.) |
| 4084 | 401 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
402 \begin{verbatim} |
| 4084 | 403 """Some documentation. |
| 404 """ | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
405 \end{verbatim} |
| 7809 | 406 |
| 4084 | 407 Using the interpreter to take a look at the parse tree, we find a |
| 408 bewildering mass of numbers and parentheses, with the documentation | |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
409 buried deep in nested tuples. |
| 3198 | 410 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
411 \begin{verbatim} |
| 4084 | 412 >>> import parser |
| 413 >>> import pprint | |
| 414 >>> ast = parser.suite(open('docstring.py').read()) | |
| 11160 | 415 >>> tup = ast.totuple() |
| 4084 | 416 >>> pprint.pprint(tup) |
| 417 (257, | |
| 418 (264, | |
| 419 (265, | |
| 420 (266, | |
| 421 (267, | |
| 422 (307, | |
| 423 (287, | |
| 424 (288, | |
| 425 (289, | |
| 426 (290, | |
| 427 (292, | |
| 428 (293, | |
| 429 (294, | |
| 430 (295, | |
| 431 (296, | |
| 432 (297, | |
| 433 (298, | |
| 434 (299, | |
| 435 (300, (3, '"""Some documentation.\012"""'))))))))))))))))), | |
| 436 (4, ''))), | |
| 437 (4, ''), | |
| 438 (0, '')) | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
439 \end{verbatim} |
| 7809 | 440 |
| 4084 | 441 The numbers at the first element of each node in the tree are the node |
| 442 types; they map directly to terminal and non-terminal symbols in the | |
| 443 grammar. Unfortunately, they are represented as integers in the | |
| 444 internal representation, and the Python structures generated do not | |
|
10703
e4e800a78d8d
Make internal module references hyperlinks wherever it makes sense.
Fred Drake <fdrake@acm.org>
parents:
10245
diff
changeset
|
445 change that. However, the \refmodule{symbol} and \refmodule{token} modules |
| 4084 | 446 provide symbolic names for the node types and dictionaries which map |
| 447 from the integers to the symbolic names for the node types. | |
| 3198 | 448 |
| 4084 | 449 In the output presented above, the outermost tuple contains four |
| 450 elements: the integer \code{257} and three additional tuples. Node | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
451 type \code{257} has the symbolic name \constant{file_input}. Each of |
| 4084 | 452 these inner tuples contains an integer as the first element; these |
| 453 integers, \code{264}, \code{4}, and \code{0}, represent the node types | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
454 \constant{stmt}, \constant{NEWLINE}, and \constant{ENDMARKER}, |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
455 respectively. |
| 4084 | 456 Note that these values may change depending on the version of Python |
| 457 you are using; consult \file{symbol.py} and \file{token.py} for | |
| 458 details of the mapping. It should be fairly clear that the outermost | |
| 459 node is related primarily to the input source rather than the contents | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
460 of the file, and may be disregarded for the moment. The \constant{stmt} |
| 4084 | 461 node is much more interesting. In particular, all docstrings are |
| 462 found in subtrees which are formed exactly as this node is formed, | |
| 463 with the only difference being the string itself. The association | |
| 464 between the docstring in a similar tree and the defined entity (class, | |
| 465 function, or module) which it describes is given by the position of | |
| 466 the docstring subtree within the tree defining the described | |
| 467 structure. | |
| 3198 | 468 |
| 4084 | 469 By replacing the actual docstring with something to signify a variable |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
470 component of the tree, we allow a simple pattern matching approach to |
|
12223
e82e82b3b9ad
Merged changes from the 1.5.2p2 release.
Fred Drake <fdrake@acm.org>
parents:
11463
diff
changeset
|
471 check any given subtree for equivalence to the general pattern for |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
472 docstrings. Since the example demonstrates information extraction, we |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
473 can safely require that the tree be in tuple form rather than list |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
474 form, allowing a simple variable representation to be |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
475 \code{['variable_name']}. A simple recursive function can implement |
| 4084 | 476 the pattern matching, returning a boolean and a dictionary of variable |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
477 name to value mappings. (See file \file{example.py}.) |
| 4084 | 478 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
479 \begin{verbatim} |
| 4084 | 480 from types import ListType, TupleType |
| 481 | |
| 482 def match(pattern, data, vars=None): | |
| 483 if vars is None: | |
| 484 vars = {} | |
| 485 if type(pattern) is ListType: | |
| 486 vars[pattern[0]] = data | |
| 487 return 1, vars | |
| 488 if type(pattern) is not TupleType: | |
| 489 return (pattern == data), vars | |
| 490 if len(data) != len(pattern): | |
| 491 return 0, vars | |
| 492 for pattern, data in map(None, pattern, data): | |
| 493 same, vars = match(pattern, data, vars) | |
| 494 if not same: | |
| 495 break | |
| 496 return same, vars | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
497 \end{verbatim} |
| 7809 | 498 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
499 Using this simple representation for syntactic variables and the symbolic |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
500 node types, the pattern for the candidate docstring subtrees becomes |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
501 fairly readable. (See file \file{example.py}.) |
| 3198 | 502 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
503 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
504 import symbol |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
505 import token |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
506 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
507 DOCSTRING_STMT_PATTERN = ( |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
508 symbol.stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
509 (symbol.simple_stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
510 (symbol.small_stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
511 (symbol.expr_stmt, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
512 (symbol.testlist, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
513 (symbol.test, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
514 (symbol.and_test, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
515 (symbol.not_test, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
516 (symbol.comparison, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
517 (symbol.expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
518 (symbol.xor_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
519 (symbol.and_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
520 (symbol.shift_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
521 (symbol.arith_expr, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
522 (symbol.term, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
523 (symbol.factor, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
524 (symbol.power, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
525 (symbol.atom, |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
526 (token.STRING, ['docstring']) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
527 )))))))))))))))), |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
528 (token.NEWLINE, '') |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
529 )) |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
530 \end{verbatim} |
| 7809 | 531 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
532 Using the \function{match()} function with this pattern, extracting the |
| 4084 | 533 module docstring from the parse tree created previously is easy: |
| 534 | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
535 \begin{verbatim} |
| 4084 | 536 >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1]) |
| 537 >>> found | |
| 538 1 | |
| 539 >>> vars | |
| 540 {'docstring': '"""Some documentation.\012"""'} | |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
541 \end{verbatim} |
| 7809 | 542 |
| 4084 | 543 Once specific data can be extracted from a location where it is |
| 544 expected, the question of where information can be expected | |
| 545 needs to be answered. When dealing with docstrings, the answer is | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
546 fairly simple: the docstring is the first \constant{stmt} node in a code |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
547 block (\constant{file_input} or \constant{suite} node types). A module |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
548 consists of a single \constant{file_input} node, and class and function |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
549 definitions each contain exactly one \constant{suite} node. Classes and |
| 4084 | 550 functions are readily identified as subtrees of code block nodes which |
| 551 start with \code{(stmt, (compound_stmt, (classdef, ...} or | |
| 552 \code{(stmt, (compound_stmt, (funcdef, ...}. Note that these subtrees | |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
553 cannot be matched by \function{match()} since it does not support multiple |
| 4084 | 554 sibling nodes to match without regard to number. A more elaborate |
| 555 matching function could be used to overcome this limitation, but this | |
| 556 is sufficient for the example. | |
| 557 | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
558 Given the ability to determine whether a statement might be a |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
559 docstring and extract the actual string from the statement, some work |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
560 needs to be performed to walk the parse tree for an entire module and |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
561 extract information about the names defined in each context of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
562 module and associate any docstrings with the names. The code to |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
563 perform this work is not complicated, but bears some explanation. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
564 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
565 The public interface to the classes is straightforward and should |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
566 probably be somewhat more flexible. Each ``major'' block of the |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
567 module is described by an object providing several methods for inquiry |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
568 and a constructor which accepts at least the subtree of the complete |
|
7363
f4a4723b9553
Separate the parse, symbol, and token documentation into separate files.
Fred Drake <fdrake@acm.org>
parents:
7277
diff
changeset
|
569 parse tree which it represents. The \class{ModuleInfo} constructor |
|
f4a4723b9553
Separate the parse, symbol, and token documentation into separate files.
Fred Drake <fdrake@acm.org>
parents:
7277
diff
changeset
|
570 accepts an optional \var{name} parameter since it cannot |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
571 otherwise determine the name of the module. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
572 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
573 The public classes include \class{ClassInfo}, \class{FunctionInfo}, |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
574 and \class{ModuleInfo}. All objects provide the |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
575 methods \method{get_name()}, \method{get_docstring()}, |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
576 \method{get_class_names()}, and \method{get_class_info()}. The |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
577 \class{ClassInfo} objects support \method{get_method_names()} and |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
578 \method{get_method_info()} while the other classes provide |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
579 \method{get_function_names()} and \method{get_function_info()}. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
580 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
581 Within each of the forms of code block that the public classes |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
582 represent, most of the required information is in the same form and is |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
583 accessed in the same way, with classes having the distinction that |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
584 functions defined at the top level are referred to as ``methods.'' |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
585 Since the difference in nomenclature reflects a real semantic |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
586 distinction from functions defined outside of a class, the |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
587 implementation needs to maintain the distinction. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
588 Hence, most of the functionality of the public classes can be |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
589 implemented in a common base class, \class{SuiteInfoBase}, with the |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
590 accessors for function and method information provided elsewhere. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
591 Note that there is only one class which represents function and method |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
592 information; this parallels the use of the \keyword{def} statement to |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
593 define both types of elements. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
594 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
595 Most of the accessor functions are declared in \class{SuiteInfoBase} |
|
13625
fdfd1049f36e
Rob W. W. Hooft's spelling fixes for the Library Reference. I hope
Thomas Wouters <thomas@python.org>
parents:
12640
diff
changeset
|
596 and do not need to be overridden by subclasses. More importantly, the |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
597 extraction of most information from a parse tree is handled through a |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
598 method called by the \class{SuiteInfoBase} constructor. The example |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
599 code for most of the classes is clear when read alongside the formal |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
600 grammar, but the method which recursively creates new information |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
601 objects requires further examination. Here is the relevant part of |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
602 the \class{SuiteInfoBase} definition from \file{example.py}: |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
603 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
604 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
605 class SuiteInfoBase: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
606 _docstring = '' |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
607 _name = '' |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
608 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
609 def __init__(self, tree = None): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
610 self._class_info = {} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
611 self._function_info = {} |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
612 if tree: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
613 self._extract_info(tree) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
614 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
615 def _extract_info(self, tree): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
616 # extract docstring |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
617 if len(tree) == 2: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
618 found, vars = match(DOCSTRING_STMT_PATTERN[1], tree[1]) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
619 else: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
620 found, vars = match(DOCSTRING_STMT_PATTERN, tree[3]) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
621 if found: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
622 self._docstring = eval(vars['docstring']) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
623 # discover inner definitions |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
624 for node in tree[1:]: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
625 found, vars = match(COMPOUND_STMT_PATTERN, node) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
626 if found: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
627 cstmt = vars['compound'] |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
628 if cstmt[0] == symbol.funcdef: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
629 name = cstmt[2][1] |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
630 self._function_info[name] = FunctionInfo(cstmt) |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
631 elif cstmt[0] == symbol.classdef: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
632 name = cstmt[2][1] |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
633 self._class_info[name] = ClassInfo(cstmt) |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
634 \end{verbatim} |
| 7809 | 635 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
636 After initializing some internal state, the constructor calls the |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
637 \method{_extract_info()} method. This method performs the bulk of the |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
638 information extraction which takes place in the entire example. The |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
639 extraction has two distinct phases: the location of the docstring for |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
640 the parse tree passed in, and the discovery of additional definitions |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
641 within the code block represented by the parse tree. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
642 |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
643 The initial \keyword{if} test determines whether the nested suite is of |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
644 the ``short form'' or the ``long form.'' The short form is used when |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
645 the code block is on the same line as the definition of the code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
646 block, as in |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
647 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
648 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
649 def square(x): "Square an argument."; return x ** 2 |
|
7050
90edcb38fb00
Added \label{} to allow module references.
Fred Drake <fdrake@acm.org>
parents:
6648
diff
changeset
|
650 \end{verbatim} |
| 7809 | 651 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
652 while the long form uses an indented block and allows nested |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
653 definitions: |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
654 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
655 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
656 def make_power(exp): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
657 "Make a function that raises an argument to the exponent `exp'." |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
658 def raiser(x, y=exp): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
659 return x ** y |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
660 return raiser |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
661 \end{verbatim} |
| 7809 | 662 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
663 When the short form is used, the code block may contain a docstring as |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
664 the first, and possibly only, \constant{small_stmt} element. The |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
665 extraction of such a docstring is slightly different and requires only |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
666 a portion of the complete pattern used in the more common case. As |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
667 implemented, the docstring will only be found if there is only |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
668 one \constant{small_stmt} node in the \constant{simple_stmt} node. |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
669 Since most functions and methods which use the short form do not |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
670 provide a docstring, this may be considered sufficient. The |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
671 extraction of the docstring proceeds using the \function{match()} function |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
672 as described above, and the value of the docstring is stored as an |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
673 attribute of the \class{SuiteInfoBase} object. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
674 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
675 After docstring extraction, a simple definition discovery |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
676 algorithm operates on the \constant{stmt} nodes of the |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
677 \constant{suite} node. The special case of the short form is not |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
678 tested; since there are no \constant{stmt} nodes in the short form, |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
679 the algorithm will silently skip the single \constant{simple_stmt} |
|
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
680 node and correctly not discover any nested definitions. |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
681 |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
682 Each statement in the code block is categorized as |
|
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
683 a class definition, function or method definition, or |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
684 something else. For the definition statements, the name of the |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
685 element defined is extracted and a representation object |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
686 appropriate to the definition is created with the defining subtree |
|
13625
fdfd1049f36e
Rob W. W. Hooft's spelling fixes for the Library Reference. I hope
Thomas Wouters <thomas@python.org>
parents:
12640
diff
changeset
|
687 passed as an argument to the constructor. The representation objects |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
688 are stored in instance variables and may be retrieved by name using |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
689 the appropriate accessor methods. |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
690 |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
691 The public classes provide any accessors required which are more |
|
7223
9dde07a593d8
Add some index entries; switch to logical markup.
Fred Drake <fdrake@acm.org>
parents:
7050
diff
changeset
|
692 specific than those provided by the \class{SuiteInfoBase} class, but |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
693 the real extraction algorithm remains common to all forms of code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
694 blocks. A high-level function can be used to extract the complete set |
|
4286
db35d2b93eba
(libparser.tex): Revised parser module documentation; improved logical
Fred Drake <fdrake@acm.org>
parents:
4132
diff
changeset
|
695 of information from a source file. (See file \file{example.py}.) |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
696 |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
697 \begin{verbatim} |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
698 def get_docs(fileName): |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
699 import os |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
700 import parser |
| 11160 | 701 |
| 702 source = open(fileName).read() | |
| 703 basename = os.path.basename(os.path.splitext(fileName)[0]) | |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
704 ast = parser.suite(source) |
| 11160 | 705 return ModuleInfo(ast.totuple(), basename) |
|
7277
bf4e790bd61a
Remove all \bcode / \ecode cruft; this is no longer needed. See previous
Fred Drake <fdrake@acm.org>
parents:
7223
diff
changeset
|
706 \end{verbatim} |
| 7809 | 707 |
|
4132
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
708 This provides an easy-to-use interface to the documentation of a |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
709 module. If information is required which is not extracted by the code |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
710 of this example, the code may be extended at clearly defined points to |
|
c18d52e06238
Final set of changes by Fred before 1.4beta3
Guido van Rossum <guido@python.org>
parents:
4084
diff
changeset
|
711 provide additional capabilities. |