Mercurial > cpython-lru_cache-weakref

annotate Doc/library/parser.rst @ 58925:eecf1bef052d 2.6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merged revisions 78207 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78207 | ezio.melotti | 2010-02-17 01:26:09 +0200 (Wed, 17 Feb 2010) | 1 line #7930: fix stripid ........
author Ezio Melotti <ezio.melotti@gmail.com>
date Tue, 16 Feb 2010 23:29:44 +0000
parents ee9ced92843a
children ff734f225b47
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
1
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
2 :mod:`parser` --- Access Python parse trees
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
3 ===========================================
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
4
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
5 .. module:: parser
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
6 :synopsis: Access parse trees for Python source code.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
7 .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
8 .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
9
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
10
44383
3625234bf3d2 Some cleanup in the docs.
Georg Brandl <georg@python.org>
parents: 43625
diff changeset
11 .. Copyright 1995 Virginia Polytechnic Institute and State University and Fred
3625234bf3d2 Some cleanup in the docs.
Georg Brandl <georg@python.org>
parents: 43625
diff changeset
12 L. Drake, Jr. This copyright notice must be distributed on all copies, but
3625234bf3d2 Some cleanup in the docs.
Georg Brandl <georg@python.org>
parents: 43625
diff changeset
13 this document otherwise may be distributed as part of the Python
3625234bf3d2 Some cleanup in the docs.
Georg Brandl <georg@python.org>
parents: 43625
diff changeset
14 distribution. No fee may be charged for this document in any representation,
3625234bf3d2 Some cleanup in the docs.
Georg Brandl <georg@python.org>
parents: 43625
diff changeset
15 either on paper or electronically. This restriction does not affect other
3625234bf3d2 Some cleanup in the docs.
Georg Brandl <georg@python.org>
parents: 43625
diff changeset
16 elements in a distributed package in any way.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
17
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
18 .. index:: single: parsing; Python source code
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
19
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
20 The :mod:`parser` module provides an interface to Python's internal parser and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
21 byte-code compiler. The primary purpose for this interface is to allow Python
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
22 code to edit the parse tree of a Python expression and create executable code
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
23 from this. This is better than trying to parse and modify an arbitrary Python
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
24 code fragment as a string because parsing is performed in a manner identical to
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
25 the code forming the application. It is also faster.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
26
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
27 .. note::
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
28
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
29 From Python 2.5 onward, it's much more convenient to cut in at the Abstract
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
30 Syntax Tree (AST) generation and compilation stage, using the :mod:`ast`
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
31 module.
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
32
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
33 The :mod:`parser` module exports the names documented here also with "st"
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
34 replaced by "ast"; this is a legacy from the time when there was no other
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
35 AST and has nothing to do with the AST found in Python 2.5. This is also the
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
36 reason for the functions' keyword arguments being called *ast*, not *st*.
48056
0441d30d672b 3k-warn about parser's "ast" aliases.
Georg Brandl <georg@python.org>
parents: 47513
diff changeset
37 The "ast" functions will be removed in Python 3.0.
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
38
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
39 There are a few things to note about this module which are important to making
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
40 use of the data structures created. This is not a tutorial on editing the parse
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
41 trees for Python code, but some examples of using the :mod:`parser` module are
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
42 presented.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
43
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
44 Most importantly, a good understanding of the Python grammar processed by the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
45 internal parser is required. For full information on the language syntax, refer
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
46 to :ref:`reference-index`. The parser
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
47 itself is created from a grammar specification defined in the file
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
48 :file:`Grammar/Grammar` in the standard Python distribution. The parse trees
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
49 stored in the ST objects created by this module are the actual output from the
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
50 internal parser when created by the :func:`expr` or :func:`suite` functions,
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
51 described below. The ST objects created by :func:`sequence2st` faithfully
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
52 simulate those structures. Be aware that the values of the sequences which are
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
53 considered "correct" will vary from one version of Python to another as the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
54 formal grammar for the language is revised. However, transporting code from one
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
55 Python version to another as source text will always allow correct parse trees
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
56 to be created in the target version, with the only restriction being that
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
57 migrating to an older version of the interpreter will not support more recent
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
58 language constructs. The parse trees are not typically compatible from one
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
59 version to another, whereas source code has always been forward-compatible.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
60
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
61 Each element of the sequences returned by :func:`st2list` or :func:`st2tuple`
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
62 has a simple form. Sequences representing non-terminal elements in the grammar
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
63 always have a length greater than one. The first element is an integer which
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
64 identifies a production in the grammar. These integers are given symbolic names
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
65 in the C header file :file:`Include/graminit.h` and the Python module
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
66 :mod:`symbol`. Each additional element of the sequence represents a component
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
67 of the production as recognized in the input string: these are always sequences
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
68 which have the same form as the parent. An important aspect of this structure
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
69 which should be noted is that keywords used to identify the parent node type,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
70 such as the keyword :keyword:`if` in an :const:`if_stmt`, are included in the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
71 node tree without any special treatment. For example, the :keyword:`if` keyword
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
72 is represented by the tuple ``(1, 'if')``, where ``1`` is the numeric value
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
73 associated with all :const:`NAME` tokens, including variable and function names
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
74 defined by the user. In an alternate form returned when line number information
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
75 is requested, the same token might be represented as ``(1, 'if', 12)``, where
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
76 the ``12`` represents the line number at which the terminal symbol was found.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
77
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
78 Terminal elements are represented in much the same way, but without any child
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
79 elements and the addition of the source text which was identified. The example
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
80 of the :keyword:`if` keyword above is representative. The various types of
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
81 terminal symbols are defined in the C header file :file:`Include/token.h` and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
82 the Python module :mod:`token`.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
83
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
84 The ST objects are not required to support the functionality of this module,
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
85 but are provided for three purposes: to allow an application to amortize the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
86 cost of processing complex parse trees, to provide a parse tree representation
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
87 which conserves memory space when compared to the Python list or tuple
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
88 representation, and to ease the creation of additional modules in C which
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
89 manipulate parse trees. A simple "wrapper" class may be created in Python to
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
90 hide the use of ST objects.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
91
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
92 The :mod:`parser` module defines functions for a few distinct purposes. The
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
93 most important purposes are to create ST objects and to convert ST objects to
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
94 other representations such as parse trees and compiled code objects, but there
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
95 are also functions which serve to query the type of parse tree represented by an
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
96 ST object.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
97
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
98
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
99 .. seealso::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
100
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
101 Module :mod:`symbol`
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
102 Useful constants representing internal nodes of the parse tree.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
103
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
104 Module :mod:`token`
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
105 Useful constants representing leaf nodes of the parse tree and functions for
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
106 testing node values.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
107
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
108
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
109 .. _creating-sts:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
110
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
111 Creating ST Objects
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
112 -------------------
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
113
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
114 ST objects may be created from source code or from a parse tree. When creating
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
115 an ST object from source, different functions are used to create the ``'eval'``
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
116 and ``'exec'`` forms.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
117
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
118
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
119 .. function:: expr(source)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
120
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
121 The :func:`expr` function parses the parameter *source* as if it were an input
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
122 to ``compile(source, 'file.py', 'eval')``. If the parse succeeds, an ST object
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
123 is created to hold the internal parse tree representation, otherwise an
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
124 appropriate exception is thrown.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
125
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
126
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
127 .. function:: suite(source)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
128
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
129 The :func:`suite` function parses the parameter *source* as if it were an input
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
130 to ``compile(source, 'file.py', 'exec')``. If the parse succeeds, an ST object
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
131 is created to hold the internal parse tree representation, otherwise an
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
132 appropriate exception is thrown.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
133
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
134
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
135 .. function:: sequence2st(sequence)
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
136
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
137 This function accepts a parse tree represented as a sequence and builds an
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
138 internal representation if possible. If it can validate that the tree conforms
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
139 to the Python grammar and all nodes are valid node types in the host version of
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
140 Python, an ST object is created from the internal representation and returned
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
141 to the called. If there is a problem creating the internal representation, or
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
142 if the tree cannot be validated, a :exc:`ParserError` exception is thrown. An
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
143 ST object created this way should not be assumed to compile correctly; normal
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
144 exceptions thrown by compilation may still be initiated when the ST object is
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
145 passed to :func:`compilest`. This may indicate problems not related to syntax
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
146 (such as a :exc:`MemoryError` exception), but may also be due to constructs such
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
147 as the result of parsing ``del f(0)``, which escapes the Python parser but is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
148 checked by the bytecode compiler.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
149
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
150 Sequences representing terminal tokens may be represented as either two-element
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
151 lists of the form ``(1, 'name')`` or as three-element lists of the form ``(1,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
152 'name', 56)``. If the third element is present, it is assumed to be a valid
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
153 line number. The line number may be specified for any subset of the terminal
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
154 symbols in the input tree.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
155
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
156
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
157 .. function:: tuple2st(sequence)
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
158
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
159 This is the same function as :func:`sequence2st`. This entry point is
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
160 maintained for backward compatibility.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
161
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
162
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
163 .. _converting-sts:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
164
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
165 Converting ST Objects
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
166 ---------------------
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
167
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
168 ST objects, regardless of the input used to create them, may be converted to
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
169 parse trees represented as list- or tuple- trees, or may be compiled into
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
170 executable code objects. Parse trees may be extracted with or without line
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
171 numbering information.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
172
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
173
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
174 .. function:: st2list(ast[, line_info])
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
175
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
176 This function accepts an ST object from the caller in *ast* and returns a
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
177 Python list representing the equivalent parse tree. The resulting list
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
178 representation can be used for inspection or the creation of a new parse tree in
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
179 list form. This function does not fail so long as memory is available to build
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
180 the list representation. If the parse tree will only be used for inspection,
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
181 :func:`st2tuple` should be used instead to reduce memory consumption and
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
182 fragmentation. When the list representation is required, this function is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
183 significantly faster than retrieving a tuple representation and converting that
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
184 to nested lists.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
185
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
186 If *line_info* is true, line number information will be included for all
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
187 terminal tokens as a third element of the list representing the token. Note
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
188 that the line number provided specifies the line on which the token *ends*.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
189 This information is omitted if the flag is false or omitted.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
190
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
191
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
192 .. function:: st2tuple(ast[, line_info])
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
193
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
194 This function accepts an ST object from the caller in *ast* and returns a
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
195 Python tuple representing the equivalent parse tree. Other than returning a
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
196 tuple instead of a list, this function is identical to :func:`st2list`.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
197
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
198 If *line_info* is true, line number information will be included for all
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
199 terminal tokens as a third element of the list representing the token. This
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
200 information is omitted if the flag is false or omitted.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
201
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
202
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
203 .. function:: compilest(ast[, filename='<syntax-tree>'])
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
204
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
205 .. index:: builtin: eval
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
206
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
207 The Python byte compiler can be invoked on an ST object to produce code objects
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
208 which can be used as part of an :keyword:`exec` statement or a call to the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
209 built-in :func:`eval` function. This function provides the interface to the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
210 compiler, passing the internal parse tree from *ast* to the parser, using the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
211 source file name specified by the *filename* parameter. The default value
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
212 supplied for *filename* indicates that the source was an ST object.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
213
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
214 Compiling an ST object may result in exceptions related to compilation; an
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
215 example would be a :exc:`SyntaxError` caused by the parse tree for ``del f(0)``:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
216 this statement is considered legal within the formal grammar for Python but is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
217 not a legal language construct. The :exc:`SyntaxError` raised for this
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
218 condition is actually generated by the Python byte-compiler normally, which is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
219 why it can be raised at this point by the :mod:`parser` module. Most causes of
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
220 compilation failure can be diagnosed programmatically by inspection of the parse
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
221 tree.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
222
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
223
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
224 .. _querying-sts:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
225
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
226 Queries on ST Objects
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
227 ---------------------
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
228
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
229 Two functions are provided which allow an application to determine if an ST was
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
230 created as an expression or a suite. Neither of these functions can be used to
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
231 determine if an ST was created from source code via :func:`expr` or
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
232 :func:`suite` or from a parse tree via :func:`sequence2st`.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
233
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
234
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
235 .. function:: isexpr(ast)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
236
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
237 .. index:: builtin: compile
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
238
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
239 When *ast* represents an ``'eval'`` form, this function returns true, otherwise
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
240 it returns false. This is useful, since code objects normally cannot be queried
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
241 for this information using existing built-in functions. Note that the code
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
242 objects created by :func:`compilest` cannot be queried like this either, and
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
243 are identical to those created by the built-in :func:`compile` function.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
244
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
245
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
246 .. function:: issuite(ast)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
247
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
248 This function mirrors :func:`isexpr` in that it reports whether an ST object
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
249 represents an ``'exec'`` form, commonly known as a "suite." It is not safe to
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
250 assume that this function is equivalent to ``not isexpr(ast)``, as additional
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
251 syntactic fragments may be supported in the future.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
252
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
253
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
254 .. _st-errors:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
255
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
256 Exceptions and Error Handling
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
257 -----------------------------
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
258
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
259 The parser module defines a single exception, but may also pass other built-in
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
260 exceptions from other portions of the Python runtime environment. See each
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
261 function for information about the exceptions it can raise.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
262
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
263
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
264 .. exception:: ParserError
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
265
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
266 Exception raised when a failure occurs within the parser module. This is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
267 generally produced for validation failures rather than the built in
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
268 :exc:`SyntaxError` thrown during normal parsing. The exception argument is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
269 either a string describing the reason of the failure or a tuple containing a
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
270 sequence causing the failure from a parse tree passed to :func:`sequence2st`
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
271 and an explanatory string. Calls to :func:`sequence2st` need to be able to
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
272 handle either type of exception, while calls to other functions in the module
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
273 will only need to be aware of the simple string values.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
274
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
275 Note that the functions :func:`compilest`, :func:`expr`, and :func:`suite` may
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
276 throw exceptions which are normally thrown by the parsing and compilation
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
277 process. These include the built in exceptions :exc:`MemoryError`,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
278 :exc:`OverflowError`, :exc:`SyntaxError`, and :exc:`SystemError`. In these
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
279 cases, these exceptions carry all the meaning normally associated with them.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
280 Refer to the descriptions of each function for detailed information.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
281
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
282
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
283 .. _st-objects:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
284
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
285 ST Objects
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
286 ----------
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
287
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
288 Ordered and equality comparisons are supported between ST objects. Pickling of
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
289 ST objects (using the :mod:`pickle` module) is also supported.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
290
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
291
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
292 .. data:: STType
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
293
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
294 The type of the objects returned by :func:`expr`, :func:`suite` and
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
295 :func:`sequence2st`.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
296
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
297 ST objects have the following methods:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
298
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
299
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
300 .. method:: ST.compile([filename])
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
301
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
302 Same as ``compilest(st, filename)``.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
303
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
304
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
305 .. method:: ST.isexpr()
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
306
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
307 Same as ``isexpr(st)``.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
308
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
309
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
310 .. method:: ST.issuite()
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
311
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
312 Same as ``issuite(st)``.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
313
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
314
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
315 .. method:: ST.tolist([line_info])
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
316
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
317 Same as ``st2list(st, line_info)``.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
318
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
319
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
320 .. method:: ST.totuple([line_info])
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
321
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
322 Same as ``st2tuple(st, line_info)``.
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
323
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
324
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
325 .. _st-examples:
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
326
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
327 Examples
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
328 --------
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
329
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
330 .. index:: builtin: compile
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
331
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
332 The parser modules allows operations to be performed on the parse tree of Python
43625
a57635100dc3 Unify "byte code" to "bytecode". Also sprinkle :term: markup for it.
Georg Brandl <georg@python.org>
parents: 42603
diff changeset
333 source code before the :term:`bytecode` is generated, and provides for inspection of the
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
334 parse tree for information gathering purposes. Two examples are presented. The
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
335 simple example demonstrates emulation of the :func:`compile` built-in function
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
336 and the complex example shows the use of a parse tree for information discovery.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
337
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
338
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
339 Emulation of :func:`compile`
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
340 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
341
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
342 While many useful operations may take place between parsing and bytecode
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
343 generation, the simplest operation is to do nothing. For this purpose, using
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
344 the :mod:`parser` module to produce an intermediate data structure is equivalent
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
345 to the code ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
346
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
347 >>> code = compile('a + 5', 'file.py', 'eval')
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
348 >>> a = 5
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
349 >>> eval(code)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
350 10
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
351
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
352 The equivalent operation using the :mod:`parser` module is somewhat longer, and
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
353 allows the intermediate internal parse tree to be retained as an ST object::
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
354
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
355 >>> import parser
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
356 >>> st = parser.expr('a + 5')
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
357 >>> code = st.compile('file.py')
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
358 >>> a = 5
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
359 >>> eval(code)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
360 10
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
361
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
362 An application which needs both ST and code objects can package this code into
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
363 readily available functions::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
364
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
365 import parser
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
366
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
367 def load_suite(source_string):
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
368 st = parser.suite(source_string)
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
369 return st, st.compile()
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
370
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
371 def load_expression(source_string):
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
372 st = parser.expr(source_string)
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
373 return st, st.compile()
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
374
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
375
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
376 Information Discovery
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
377 ^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
378
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
379 .. index::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
380 single: string; documentation
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
381 single: docstrings
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
382
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
383 Some applications benefit from direct access to the parse tree. The remainder
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
384 of this section demonstrates how the parse tree provides access to module
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
385 documentation defined in docstrings without requiring that the code being
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
386 examined be loaded into a running interpreter via :keyword:`import`. This can
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
387 be very useful for performing analyses of untrusted code.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
388
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
389 Generally, the example will demonstrate how the parse tree may be traversed to
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
390 distill interesting information. Two functions and a set of classes are
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
391 developed which provide programmatic access to high level function and class
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
392 definitions provided by a module. The classes extract information from the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
393 parse tree and provide access to the information at a useful semantic level, one
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
394 function provides a simple low-level pattern matching capability, and the other
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
395 function defines a high-level interface to the classes by handling file
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
396 operations on behalf of the caller. All source files mentioned here which are
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
397 not part of the Python installation are located in the :file:`Demo/parser/`
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
398 directory of the distribution.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
399
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
400 The dynamic nature of Python allows the programmer a great deal of flexibility,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
401 but most modules need only a limited measure of this when defining classes,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
402 functions, and methods. In this example, the only definitions that will be
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
403 considered are those which are defined in the top level of their context, e.g.,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
404 a function defined by a :keyword:`def` statement at column zero of a module, but
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
405 not a function defined within a branch of an :keyword:`if` ... :keyword:`else`
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
406 construct, though there are some good reasons for doing so in some situations.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
407 Nesting of definitions will be handled by the code developed in the example.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
408
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
409 To construct the upper-level extraction methods, we need to know what the parse
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
410 tree structure looks like and how much of it we actually need to be concerned
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
411 about. Python uses a moderately deep parse tree so there are a large number of
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
412 intermediate nodes. It is important to read and understand the formal grammar
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
413 used by Python. This is specified in the file :file:`Grammar/Grammar` in the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
414 distribution. Consider the simplest case of interest when searching for
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
415 docstrings: a module consisting of a docstring and nothing else. (See file
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
416 :file:`docstring.py`.) ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
417
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
418 """Some documentation.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
419 """
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
420
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
421 Using the interpreter to take a look at the parse tree, we find a bewildering
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
422 mass of numbers and parentheses, with the documentation buried deep in nested
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
423 tuples. ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
424
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
425 >>> import parser
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
426 >>> import pprint
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
427 >>> st = parser.suite(open('docstring.py').read())
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
428 >>> tup = st.totuple()
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
429 >>> pprint.pprint(tup)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
430 (257,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
431 (264,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
432 (265,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
433 (266,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
434 (267,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
435 (307,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
436 (287,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
437 (288,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
438 (289,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
439 (290,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
440 (292,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
441 (293,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
442 (294,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
443 (295,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
444 (296,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
445 (297,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
446 (298,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
447 (299,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
448 (300, (3, '"""Some documentation.\n"""'))))))))))))))))),
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
449 (4, ''))),
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
450 (4, ''),
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
451 (0, ''))
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
452
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
453 The numbers at the first element of each node in the tree are the node types;
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
454 they map directly to terminal and non-terminal symbols in the grammar.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
455 Unfortunately, they are represented as integers in the internal representation,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
456 and the Python structures generated do not change that. However, the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
457 :mod:`symbol` and :mod:`token` modules provide symbolic names for the node types
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
458 and dictionaries which map from the integers to the symbolic names for the node
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
459 types.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
460
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
461 In the output presented above, the outermost tuple contains four elements: the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
462 integer ``257`` and three additional tuples. Node type ``257`` has the symbolic
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
463 name :const:`file_input`. Each of these inner tuples contains an integer as the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
464 first element; these integers, ``264``, ``4``, and ``0``, represent the node
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
465 types :const:`stmt`, :const:`NEWLINE`, and :const:`ENDMARKER`, respectively.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
466 Note that these values may change depending on the version of Python you are
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
467 using; consult :file:`symbol.py` and :file:`token.py` for details of the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
468 mapping. It should be fairly clear that the outermost node is related primarily
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
469 to the input source rather than the contents of the file, and may be disregarded
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
470 for the moment. The :const:`stmt` node is much more interesting. In
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
471 particular, all docstrings are found in subtrees which are formed exactly as
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
472 this node is formed, with the only difference being the string itself. The
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
473 association between the docstring in a similar tree and the defined entity
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
474 (class, function, or module) which it describes is given by the position of the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
475 docstring subtree within the tree defining the described structure.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
476
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
477 By replacing the actual docstring with something to signify a variable component
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
478 of the tree, we allow a simple pattern matching approach to check any given
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
479 subtree for equivalence to the general pattern for docstrings. Since the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
480 example demonstrates information extraction, we can safely require that the tree
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
481 be in tuple form rather than list form, allowing a simple variable
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
482 representation to be ``['variable_name']``. A simple recursive function can
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
483 implement the pattern matching, returning a Boolean and a dictionary of variable
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
484 name to value mappings. (See file :file:`example.py`.) ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
485
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
486 from types import ListType, TupleType
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
487
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
488 def match(pattern, data, vars=None):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
489 if vars is None:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
490 vars = {}
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
491 if type(pattern) is ListType:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
492 vars[pattern[0]] = data
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
493 return 1, vars
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
494 if type(pattern) is not TupleType:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
495 return (pattern == data), vars
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
496 if len(data) != len(pattern):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
497 return 0, vars
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
498 for pattern, data in map(None, pattern, data):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
499 same, vars = match(pattern, data, vars)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
500 if not same:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
501 break
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
502 return same, vars
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
503
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
504 Using this simple representation for syntactic variables and the symbolic node
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
505 types, the pattern for the candidate docstring subtrees becomes fairly readable.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
506 (See file :file:`example.py`.) ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
507
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
508 import symbol
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
509 import token
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
510
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
511 DOCSTRING_STMT_PATTERN = (
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
512 symbol.stmt,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
513 (symbol.simple_stmt,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
514 (symbol.small_stmt,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
515 (symbol.expr_stmt,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
516 (symbol.testlist,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
517 (symbol.test,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
518 (symbol.and_test,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
519 (symbol.not_test,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
520 (symbol.comparison,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
521 (symbol.expr,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
522 (symbol.xor_expr,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
523 (symbol.and_expr,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
524 (symbol.shift_expr,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
525 (symbol.arith_expr,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
526 (symbol.term,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
527 (symbol.factor,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
528 (symbol.power,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
529 (symbol.atom,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
530 (token.STRING, ['docstring'])
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
531 )))))))))))))))),
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
532 (token.NEWLINE, '')
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
533 ))
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
534
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
535 Using the :func:`match` function with this pattern, extracting the module
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
536 docstring from the parse tree created previously is easy::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
537
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
538 >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1])
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
539 >>> found
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
540 1
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
541 >>> vars
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
542 {'docstring': '"""Some documentation.\n"""'}
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
543
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
544 Once specific data can be extracted from a location where it is expected, the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
545 question of where information can be expected needs to be answered. When
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
546 dealing with docstrings, the answer is fairly simple: the docstring is the first
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
547 :const:`stmt` node in a code block (:const:`file_input` or :const:`suite` node
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
548 types). A module consists of a single :const:`file_input` node, and class and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
549 function definitions each contain exactly one :const:`suite` node. Classes and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
550 functions are readily identified as subtrees of code block nodes which start
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
551 with ``(stmt, (compound_stmt, (classdef, ...`` or ``(stmt, (compound_stmt,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
552 (funcdef, ...``. Note that these subtrees cannot be matched by :func:`match`
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
553 since it does not support multiple sibling nodes to match without regard to
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
554 number. A more elaborate matching function could be used to overcome this
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
555 limitation, but this is sufficient for the example.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
556
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
557 Given the ability to determine whether a statement might be a docstring and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
558 extract the actual string from the statement, some work needs to be performed to
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
559 walk the parse tree for an entire module and extract information about the names
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
560 defined in each context of the module and associate any docstrings with the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
561 names. The code to perform this work is not complicated, but bears some
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
562 explanation.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
563
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
564 The public interface to the classes is straightforward and should probably be
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
565 somewhat more flexible. Each "major" block of the module is described by an
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
566 object providing several methods for inquiry and a constructor which accepts at
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
567 least the subtree of the complete parse tree which it represents. The
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
568 :class:`ModuleInfo` constructor accepts an optional *name* parameter since it
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
569 cannot otherwise determine the name of the module.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
570
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
571 The public classes include :class:`ClassInfo`, :class:`FunctionInfo`, and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
572 :class:`ModuleInfo`. All objects provide the methods :meth:`get_name`,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
573 :meth:`get_docstring`, :meth:`get_class_names`, and :meth:`get_class_info`. The
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
574 :class:`ClassInfo` objects support :meth:`get_method_names` and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
575 :meth:`get_method_info` while the other classes provide
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
576 :meth:`get_function_names` and :meth:`get_function_info`.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
577
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
578 Within each of the forms of code block that the public classes represent, most
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
579 of the required information is in the same form and is accessed in the same way,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
580 with classes having the distinction that functions defined at the top level are
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
581 referred to as "methods." Since the difference in nomenclature reflects a real
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
582 semantic distinction from functions defined outside of a class, the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
583 implementation needs to maintain the distinction. Hence, most of the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
584 functionality of the public classes can be implemented in a common base class,
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
585 :class:`SuiteInfoBase`, with the accessors for function and method information
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
586 provided elsewhere. Note that there is only one class which represents function
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
587 and method information; this parallels the use of the :keyword:`def` statement
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
588 to define both types of elements.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
589
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
590 Most of the accessor functions are declared in :class:`SuiteInfoBase` and do not
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
591 need to be overridden by subclasses. More importantly, the extraction of most
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
592 information from a parse tree is handled through a method called by the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
593 :class:`SuiteInfoBase` constructor. The example code for most of the classes is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
594 clear when read alongside the formal grammar, but the method which recursively
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
595 creates new information objects requires further examination. Here is the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
596 relevant part of the :class:`SuiteInfoBase` definition from :file:`example.py`::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
597
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
598 class SuiteInfoBase:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
599 _docstring = ''
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
600 _name = ''
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
601
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
602 def __init__(self, tree = None):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
603 self._class_info = {}
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
604 self._function_info = {}
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
605 if tree:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
606 self._extract_info(tree)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
607
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
608 def _extract_info(self, tree):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
609 # extract docstring
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
610 if len(tree) == 2:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
611 found, vars = match(DOCSTRING_STMT_PATTERN[1], tree[1])
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
612 else:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
613 found, vars = match(DOCSTRING_STMT_PATTERN, tree[3])
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
614 if found:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
615 self._docstring = eval(vars['docstring'])
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
616 # discover inner definitions
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
617 for node in tree[1:]:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
618 found, vars = match(COMPOUND_STMT_PATTERN, node)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
619 if found:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
620 cstmt = vars['compound']
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
621 if cstmt[0] == symbol.funcdef:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
622 name = cstmt[2][1]
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
623 self._function_info[name] = FunctionInfo(cstmt)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
624 elif cstmt[0] == symbol.classdef:
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
625 name = cstmt[2][1]
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
626 self._class_info[name] = ClassInfo(cstmt)
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
627
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
628 After initializing some internal state, the constructor calls the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
629 :meth:`_extract_info` method. This method performs the bulk of the information
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
630 extraction which takes place in the entire example. The extraction has two
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
631 distinct phases: the location of the docstring for the parse tree passed in, and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
632 the discovery of additional definitions within the code block represented by the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
633 parse tree.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
634
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
635 The initial :keyword:`if` test determines whether the nested suite is of the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
636 "short form" or the "long form." The short form is used when the code block is
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
637 on the same line as the definition of the code block, as in ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
638
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
639 def square(x): "Square an argument."; return x ** 2
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
640
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
641 while the long form uses an indented block and allows nested definitions::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
642
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
643 def make_power(exp):
50522
ee9ced92843a Merged revisions 68133-68134,68141-68142,68145-68146,68148-68149,68159-68162,68166,68171-68174,68179,68195-68196,68210,68214-68215,68217-68222 via svnmerge from
Georg Brandl <georg@python.org>
parents: 48056
diff changeset
644 "Make a function that raises an argument to the exponent `exp`."
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
645 def raiser(x, y=exp):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
646 return x ** y
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
647 return raiser
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
648
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
649 When the short form is used, the code block may contain a docstring as the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
650 first, and possibly only, :const:`small_stmt` element. The extraction of such a
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
651 docstring is slightly different and requires only a portion of the complete
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
652 pattern used in the more common case. As implemented, the docstring will only
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
653 be found if there is only one :const:`small_stmt` node in the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
654 :const:`simple_stmt` node. Since most functions and methods which use the short
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
655 form do not provide a docstring, this may be considered sufficient. The
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
656 extraction of the docstring proceeds using the :func:`match` function as
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
657 described above, and the value of the docstring is stored as an attribute of the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
658 :class:`SuiteInfoBase` object.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
659
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
660 After docstring extraction, a simple definition discovery algorithm operates on
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
661 the :const:`stmt` nodes of the :const:`suite` node. The special case of the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
662 short form is not tested; since there are no :const:`stmt` nodes in the short
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
663 form, the algorithm will silently skip the single :const:`simple_stmt` node and
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
664 correctly not discover any nested definitions.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
665
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
666 Each statement in the code block is categorized as a class definition, function
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
667 or method definition, or something else. For the definition statements, the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
668 name of the element defined is extracted and a representation object appropriate
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
669 to the definition is created with the defining subtree passed as an argument to
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
670 the constructor. The representation objects are stored in instance variables
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
671 and may be retrieved by name using the appropriate accessor methods.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
672
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
673 The public classes provide any accessors required which are more specific than
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
674 those provided by the :class:`SuiteInfoBase` class, but the real extraction
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
675 algorithm remains common to all forms of code blocks. A high-level function can
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
676 be used to extract the complete set of information from a source file. (See
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
677 file :file:`example.py`.) ::
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
678
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
679 def get_docs(fileName):
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
680 import os
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
681 import parser
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
682
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
683 source = open(fileName).read()
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
684 basename = os.path.basename(os.path.splitext(fileName)[0])
47513
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
685 st = parser.suite(source)
4f70a9de2865 Document the "st" API, to avoid confusion with the "new" AST.
Georg Brandl <georg@python.org>
parents: 44383
diff changeset
686 return ModuleInfo(st.totuple(), basename)
42603
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
687
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
688 This provides an easy-to-use interface to the documentation of a module. If
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
689 information is required which is not extracted by the code of this example, the
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
690 code may be extended at clearly defined points to provide additional
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
691 capabilities.
c62f6669ae22 Move the 2.6 reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
692