cpython-lru_cache-weakref: Doc/libparser.tex annotate

annotate Doc/libparser.tex @ 7278:59d2761ba1cf legacy-trunk

Remove obsolete pagenumbering / pagestyle cruft.

author	Fred Drake <fdrake@acm.org>
date	Fri, 13 Feb 1998 07:08:56 +0000
parents	bf4e790bd61a
children	f4a4723b9553

rev	line source
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	1 % libparser.tex
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	2 %
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	3 % Copyright 1995 Virginia Polytechnic Institute and State University
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	4 % and Fred L. Drake, Jr. This copyright notice must be distributed on
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	5 % all copies, but this document otherwise may be distributed as part
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	6 % of the Python distribution. No fee may be charged for this document
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	7 % in any representation, either on paper or electronically. This
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	8 % restriction does not affect other elements in a distributed package
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	9 % in any way.
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	10 %
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	11
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	12 \section{Built-in Module \sectcode{parser}}
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	13 \label{module-parser}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	14 \bimodindex{parser}
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	15 \index{parsing!Python source code}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	16
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	17 The \module{parser} module provides an interface to Python's internal
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	18 parser and byte-code compiler. The primary purpose for this interface
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	19 is to allow Python code to edit the parse tree of a Python expression
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	20 and create executable code from this. This is better than trying
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	21 to parse and modify an arbitrary Python code fragment as a string
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	22 because parsing is performed in a manner identical to the code
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	23 forming the application. It is also faster.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	24
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	25 There are a few things to note about this module which are important
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	26 to making use of the data structures created. This is not a tutorial
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	27 on editing the parse trees for Python code, but some examples of using
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	28 the \module{parser} module are presented.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	29
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	30 Most importantly, a good understanding of the Python grammar processed
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	31 by the internal parser is required. For full information on the
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	32 language syntax, refer to the \emph{Python Language Reference}. The
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	33 parser itself is created from a grammar specification defined in the file
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	34 \file{Grammar/Grammar} in the standard Python distribution. The parse
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	35 trees stored in the ``AST objects'' created by this module are the
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	36 actual output from the internal parser when created by the
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	37 \function{expr()} or \function{suite()} functions, described below. The AST
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	38 objects created by \function{sequence2ast()} faithfully simulate those
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	39 structures. Be aware that the values of the sequences which are
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	40 considered ``correct'' will vary from one version of Python to another
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	41 as the formal grammar for the language is revised. However,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	42 transporting code from one Python version to another as source text
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	43 will always allow correct parse trees to be created in the target
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	44 version, with the only restriction being that migrating to an older
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	45 version of the interpreter will not support more recent language
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	46 constructs. The parse trees are not typically compatible from one
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	47 version to another, whereas source code has always been
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	48 forward-compatible.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	49
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	50 Each element of the sequences returned by \function{ast2list()} or
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	51 \function{ast2tuple()} has a simple form. Sequences representing
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	52 non-terminal elements in the grammar always have a length greater than
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	53 one. The first element is an integer which identifies a production in
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	54 the grammar. These integers are given symbolic names in the C header
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	55 file \file{Include/graminit.h} and the Python module
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	56 \module{symbol}. Each additional element of the sequence represents
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	57 a component of the production as recognized in the input string: these
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	58 are always sequences which have the same form as the parent. An
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	59 important aspect of this structure which should be noted is that
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	60 keywords used to identify the parent node type, such as the keyword
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	61 \keyword{if} in an \constant{if_stmt}, are included in the node tree without
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	62 any special treatment. For example, the \keyword{if} keyword is
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	63 represented by the tuple \code{(1, 'if')}, where \code{1} is the
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	64 numeric value associated with all \code{NAME} tokens, including
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	65 variable and function names defined by the user. In an alternate form
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	66 returned when line number information is requested, the same token
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	67 might be represented as \code{(1, 'if', 12)}, where the \code{12}
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	68 represents the line number at which the terminal symbol was found.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	69
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	70 Terminal elements are represented in much the same way, but without
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	71 any child elements and the addition of the source text which was
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	72 identified. The example of the \keyword{if} keyword above is
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	73 representative. The various types of terminal symbols are defined in
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	74 the C header file \file{Include/token.h} and the Python module
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	75 \module{token}.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	76
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	77 The AST objects are not required to support the functionality of this
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	78 module, but are provided for three purposes: to allow an application
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	79 to amortize the cost of processing complex parse trees, to provide a
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	80 parse tree representation which conserves memory space when compared
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	81 to the Python list or tuple representation, and to ease the creation
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	82 of additional modules in C which manipulate parse trees. A simple
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	83 ``wrapper'' class may be created in Python to hide the use of AST
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	84 objects.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	85
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	86 The \module{parser} module defines functions for a few distinct
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	87 purposes. The most important purposes are to create AST objects and
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	88 to convert AST objects to other representations such as parse trees
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	89 and compiled code objects, but there are also functions which serve to
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	90 query the type of parse tree represented by an AST object.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	91
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	92 \setindexsubitem{(in module parser)}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	93
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	94
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	95 \subsection{Creating AST Objects}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	96
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	97 AST objects may be created from source code or from a parse tree.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	98 When creating an AST object from source, different functions are used
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	99 to create the \code{'eval'} and \code{'exec'} forms.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	100
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	101 \begin{funcdesc}{expr}{string}
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	102 The \function{expr()} function parses the parameter \code{\var{string}}
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	103 as if it were an input to \samp{compile(\var{string}, 'eval')}. If
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	104 the parse succeeds, an AST object is created to hold the internal
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	105 parse tree representation, otherwise an appropriate exception is
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	106 thrown.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	107 \end{funcdesc}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	108
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	109 \begin{funcdesc}{suite}{string}
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	110 The \function{suite()} function parses the parameter \code{\var{string}}
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	111 as if it were an input to \samp{compile(\var{string}, 'exec')}. If
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	112 the parse succeeds, an AST object is created to hold the internal
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	113 parse tree representation, otherwise an appropriate exception is
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	114 thrown.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	115 \end{funcdesc}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	116
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	117 \begin{funcdesc}{sequence2ast}{sequence}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	118 This function accepts a parse tree represented as a sequence and
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	119 builds an internal representation if possible. If it can validate
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	120 that the tree conforms to the Python grammar and all nodes are valid
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	121 node types in the host version of Python, an AST object is created
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	122 from the internal representation and returned to the called. If there
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	123 is a problem creating the internal representation, or if the tree
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	124 cannot be validated, a \exception{ParserError} exception is thrown. An AST
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	125 object created this way should not be assumed to compile correctly;
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	126 normal exceptions thrown by compilation may still be initiated when
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	127 the AST object is passed to \function{compileast()}. This may indicate
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	128 problems not related to syntax (such as a \exception{MemoryError}
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	129 exception), but may also be due to constructs such as the result of
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	130 parsing \code{del f(0)}, which escapes the Python parser but is
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	131 checked by the bytecode compiler.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	132
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	133 Sequences representing terminal tokens may be represented as either
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	134 two-element lists of the form \code{(1, 'name')} or as three-element
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	135 lists of the form \code{(1, 'name', 56)}. If the third element is
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	136 present, it is assumed to be a valid line number. The line number
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	137 may be specified for any subset of the terminal symbols in the input
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	138 tree.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	139 \end{funcdesc}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	140
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	141 \begin{funcdesc}{tuple2ast}{sequence}
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	142 This is the same function as \function{sequence2ast()}. This entry point
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	143 is maintained for backward compatibility.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	144 \end{funcdesc}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	145
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	146
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	147 \subsection{Converting AST Objects}
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	148
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	149 AST objects, regardless of the input used to create them, may be
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	150 converted to parse trees represented as list- or tuple- trees, or may
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	151 be compiled into executable code objects. Parse trees may be
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	152 extracted with or without line numbering information.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	153
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	154 \begin{funcdesc}{ast2list}{ast\optional{\, line_info\code{ = 0}}}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	155 This function accepts an AST object from the caller in
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	156 \code{\var{ast}} and returns a Python list representing the
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	157 equivelent parse tree. The resulting list representation can be used
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	158 for inspection or the creation of a new parse tree in list form. This
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	159 function does not fail so long as memory is available to build the
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	160 list representation. If the parse tree will only be used for
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	161 inspection, \function{ast2tuple()} should be used instead to reduce memory
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	162 consumption and fragmentation. When the list representation is
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	163 required, this function is significantly faster than retrieving a
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	164 tuple representation and converting that to nested lists.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	165
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	166 If \code{\var{line_info}} is true, line number information will be
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	167 included for all terminal tokens as a third element of the list
4591 8cdd94a3d02d (libparser.tex): Added note that the line number for a terminal token Fred Drake <fdrake@acm.org> parents: 4286 diff changeset	168 representing the token. Note that the line number provided specifies
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	169 the line on which the token \emph{ends}. This information is
4591 8cdd94a3d02d (libparser.tex): Added note that the line number for a terminal token Fred Drake <fdrake@acm.org> parents: 4286 diff changeset	170 omitted if the flag is false or omitted.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	171 \end{funcdesc}
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	172
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	173 \begin{funcdesc}{ast2tuple}{ast\optional{\, line_info\code{ = 0}}}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	174 This function accepts an AST object from the caller in
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	175 \code{\var{ast}} and returns a Python tuple representing the
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	176 equivelent parse tree. Other than returning a tuple instead of a
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	177 list, this function is identical to \function{ast2list()}.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	178
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	179 If \code{\var{line_info}} is true, line number information will be
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	180 included for all terminal tokens as a third element of the list
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	181 representing the token. This information is omitted if the flag is
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	182 false or omitted.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	183 \end{funcdesc}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	184
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	185 \begin{funcdesc}{compileast}{ast\optional{\, filename\code{ = '<ast>'}}}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	186 The Python byte compiler can be invoked on an AST object to produce
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	187 code objects which can be used as part of an \code{exec} statement or
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	188 a call to the built-in \function{eval()}\bifuncindex{eval} function.
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	189 This function provides the interface to the compiler, passing the
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	190 internal parse tree from \code{\var{ast}} to the parser, using the
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	191 source file name specified by the \code{\var{filename}} parameter.
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	192 The default value supplied for \code{\var{filename}} indicates that
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	193 the source was an AST object.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	194
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	195 Compiling an AST object may result in exceptions related to
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	196 compilation; an example would be a \exception{SyntaxError} caused by the
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	197 parse tree for \code{del f(0)}: this statement is considered legal
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	198 within the formal grammar for Python but is not a legal language
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	199 construct. The \exception{SyntaxError} raised for this condition is
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	200 actually generated by the Python byte-compiler normally, which is why
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	201 it can be raised at this point by the \module{parser} module. Most
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	202 causes of compilation failure can be diagnosed programmatically by
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	203 inspection of the parse tree.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	204 \end{funcdesc}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	205
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	206
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	207 \subsection{Queries on AST Objects}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	208
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	209 Two functions are provided which allow an application to determine if
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	210 an AST was create as an expression or a suite. Neither of these
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	211 functions can be used to determine if an AST was created from source
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	212 code via \function{expr()} or \function{suite()} or from a parse tree
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	213 via \function{sequence2ast()}.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	214
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	215 \begin{funcdesc}{isexpr}{ast}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	216 When \code{\var{ast}} represents an \code{'eval'} form, this function
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	217 returns true, otherwise it returns false. This is useful, since code
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	218 objects normally cannot be queried for this information using existing
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	219 built-in functions. Note that the code objects created by
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	220 \function{compileast()} cannot be queried like this either, and are
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	221 identical to those created by the built-in
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	222 \function{compile()}\bifuncindex{compile} function.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	223 \end{funcdesc}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	224
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	225
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	226 \begin{funcdesc}{issuite}{ast}
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	227 This function mirrors \function{isexpr()} in that it reports whether an
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	228 AST object represents an \code{'exec'} form, commonly known as a
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	229 ``suite.'' It is not safe to assume that this function is equivelent
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	230 to \samp{not isexpr(\var{ast})}, as additional syntactic fragments may
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	231 be supported in the future.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	232 \end{funcdesc}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	233
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	234
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	235 \subsection{Exceptions and Error Handling}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	236
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	237 The parser module defines a single exception, but may also pass other
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	238 built-in exceptions from other portions of the Python runtime
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	239 environment. See each function for information about the exceptions
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	240 it can raise.
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	241
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	242 \begin{excdesc}{ParserError}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	243 Exception raised when a failure occurs within the parser module. This
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	244 is generally produced for validation failures rather than the built in
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	245 \exception{SyntaxError} thrown during normal parsing.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	246 The exception argument is either a string describing the reason of the
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	247 failure or a tuple containing a sequence causing the failure from a parse
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	248 tree passed to \function{sequence2ast()} and an explanatory string. Calls to
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	249 \function{sequence2ast()} need to be able to handle either type of exception,
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	250 while calls to other functions in the module will only need to be
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	251 aware of the simple string values.
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	252 \end{excdesc}
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	253
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	254 Note that the functions \function{compileast()}, \function{expr()}, and
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	255 \function{suite()} may throw exceptions which are normally thrown by the
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	256 parsing and compilation process. These include the built in
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	257 exceptions \exception{MemoryError}, \exception{OverflowError},
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	258 \exception{SyntaxError}, and \exception{SystemError}. In these cases, these
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	259 exceptions carry all the meaning normally associated with them. Refer
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	260 to the descriptions of each function for detailed information.
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	261
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	262
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	263 \subsection{AST Objects}
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	264
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	265 AST objects returned by \function{expr()}, \function{suite()}, and
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	266 \function{sequence2ast()} have no methods of their own.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	267 Some of the functions defined which accept an AST object as their
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	268 first argument may change to object methods in the future. The type
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	269 of these objects is available as \code{ASTType} in the module.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	270
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	271 Ordered and equality comparisons are supported between AST objects.
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	272
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	273
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	274 \subsection{Examples}
4728 b218f294b726 (lib<all sorts of stuff>.tex): Fred Drake <fdrake@acm.org> parents: 4591 diff changeset	275 \nodename{AST Examples}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	276
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	277 The parser modules allows operations to be performed on the parse tree
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	278 of Python source code before the bytecode is generated, and provides
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	279 for inspection of the parse tree for information gathering purposes.
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	280 Two examples are presented. The simple example demonstrates emulation
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	281 of the \function{compile()}\bifuncindex{compile} built-in function and
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	282 the complex example shows the use of a parse tree for information
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	283 discovery.
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	284
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	285 \subsubsection{Emulation of \sectcode{compile()}}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	286
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	287 While many useful operations may take place between parsing and
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	288 bytecode generation, the simplest operation is to do nothing. For
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	289 this purpose, using the \module{parser} module to produce an
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	290 intermediate data structure is equivelent to the code
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	291
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	292 \begin{verbatim}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	293 >>> code = compile('a + 5', 'eval')
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	294 >>> a = 5
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	295 >>> eval(code)
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	296 10
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	297 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	298 %
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	299 The equivelent operation using the \module{parser} module is somewhat
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	300 longer, and allows the intermediate internal parse tree to be retained
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	301 as an AST object:
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	302
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	303 \begin{verbatim}
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	304 >>> import parser
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	305 >>> ast = parser.expr('a + 5')
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	306 >>> code = parser.compileast(ast)
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	307 >>> a = 5
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	308 >>> eval(code)
21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	309 10
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	310 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	311 %
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	312 An application which needs both AST and code objects can package this
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	313 code into readily available functions:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	314
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	315 \begin{verbatim}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	316 import parser
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	317
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	318 def load_suite(source_string):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	319 ast = parser.suite(source_string)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	320 code = parser.compileast(ast)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	321 return ast, code
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	322
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	323 def load_expression(source_string):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	324 ast = parser.expr(source_string)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	325 code = parser.compileast(ast)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	326 return ast, code
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	327 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	328 %
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	329 \subsubsection{Information Discovery}
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	330
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	331 Some applications benefit from direct access to the parse tree. The
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	332 remainder of this section demonstrates how the parse tree provides
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	333 access to module documentation defined in docstrings without requiring
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	334 that the code being examined be loaded into a running interpreter via
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	335 \keyword{import}. This can be very useful for performing analyses of
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	336 untrusted code.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	337
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	338 Generally, the example will demonstrate how the parse tree may be
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	339 traversed to distill interesting information. Two functions and a set
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	340 of classes are developed which provide programmatic access to high
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	341 level function and class definitions provided by a module. The
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	342 classes extract information from the parse tree and provide access to
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	343 the information at a useful semantic level, one function provides a
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	344 simple low-level pattern matching capability, and the other function
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	345 defines a high-level interface to the classes by handling file
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	346 operations on behalf of the caller. All source files mentioned here
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	347 which are not part of the Python installation are located in the
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	348 \file{Demo/parser/} directory of the distribution.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	349
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	350 The dynamic nature of Python allows the programmer a great deal of
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	351 flexibility, but most modules need only a limited measure of this when
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	352 defining classes, functions, and methods. In this example, the only
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	353 definitions that will be considered are those which are defined in the
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	354 top level of their context, e.g., a function defined by a \keyword{def}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	355 statement at column zero of a module, but not a function defined
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	356 within a branch of an \code{if} ... \code{else} construct, though
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	357 there are some good reasons for doing so in some situations. Nesting
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	358 of definitions will be handled by the code developed in the example.
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	359
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	360 To construct the upper-level extraction methods, we need to know what
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	361 the parse tree structure looks like and how much of it we actually
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	362 need to be concerned about. Python uses a moderately deep parse tree
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	363 so there are a large number of intermediate nodes. It is important to
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	364 read and understand the formal grammar used by Python. This is
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	365 specified in the file \file{Grammar/Grammar} in the distribution.
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	366 Consider the simplest case of interest when searching for docstrings:
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	367 a module consisting of a docstring and nothing else. (See file
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	368 \file{docstring.py}.)
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	369
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	370 \begin{verbatim}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	371 """Some documentation.
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	372 """
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	373 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	374 %
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	375 Using the interpreter to take a look at the parse tree, we find a
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	376 bewildering mass of numbers and parentheses, with the documentation
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	377 buried deep in nested tuples.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	378
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	379 \begin{verbatim}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	380 >>> import parser
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	381 >>> import pprint
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	382 >>> ast = parser.suite(open('docstring.py').read())
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	383 >>> tup = parser.ast2tuple(ast)
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	384 >>> pprint.pprint(tup)
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	385 (257,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	386 (264,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	387 (265,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	388 (266,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	389 (267,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	390 (307,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	391 (287,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	392 (288,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	393 (289,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	394 (290,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	395 (292,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	396 (293,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	397 (294,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	398 (295,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	399 (296,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	400 (297,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	401 (298,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	402 (299,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	403 (300, (3, '"""Some documentation.\012"""'))))))))))))))))),
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	404 (4, ''))),
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	405 (4, ''),
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	406 (0, ''))
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	407 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	408 %
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	409 The numbers at the first element of each node in the tree are the node
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	410 types; they map directly to terminal and non-terminal symbols in the
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	411 grammar. Unfortunately, they are represented as integers in the
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	412 internal representation, and the Python structures generated do not
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	413 change that. However, the \module{symbol} and \module{token} modules
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	414 provide symbolic names for the node types and dictionaries which map
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	415 from the integers to the symbolic names for the node types.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	416
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	417 In the output presented above, the outermost tuple contains four
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	418 elements: the integer \code{257} and three additional tuples. Node
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	419 type \code{257} has the symbolic name \constant{file_input}. Each of
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	420 these inner tuples contains an integer as the first element; these
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	421 integers, \code{264}, \code{4}, and \code{0}, represent the node types
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	422 \constant{stmt}, \constant{NEWLINE}, and \constant{ENDMARKER},
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	423 respectively.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	424 Note that these values may change depending on the version of Python
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	425 you are using; consult \file{symbol.py} and \file{token.py} for
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	426 details of the mapping. It should be fairly clear that the outermost
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	427 node is related primarily to the input source rather than the contents
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	428 of the file, and may be disregarded for the moment. The \constant{stmt}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	429 node is much more interesting. In particular, all docstrings are
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	430 found in subtrees which are formed exactly as this node is formed,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	431 with the only difference being the string itself. The association
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	432 between the docstring in a similar tree and the defined entity (class,
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	433 function, or module) which it describes is given by the position of
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	434 the docstring subtree within the tree defining the described
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	435 structure.
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	436
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	437 By replacing the actual docstring with something to signify a variable
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	438 component of the tree, we allow a simple pattern matching approach to
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	439 check any given subtree for equivelence to the general pattern for
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	440 docstrings. Since the example demonstrates information extraction, we
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	441 can safely require that the tree be in tuple form rather than list
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	442 form, allowing a simple variable representation to be
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	443 \code{['variable_name']}. A simple recursive function can implement
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	444 the pattern matching, returning a boolean and a dictionary of variable
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	445 name to value mappings. (See file \file{example.py}.)
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	446
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	447 \begin{verbatim}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	448 from types import ListType, TupleType
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	449
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	450 def match(pattern, data, vars=None):
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	451 if vars is None:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	452 vars = {}
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	453 if type(pattern) is ListType:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	454 vars[pattern[0]] = data
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	455 return 1, vars
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	456 if type(pattern) is not TupleType:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	457 return (pattern == data), vars
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	458 if len(data) != len(pattern):
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	459 return 0, vars
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	460 for pattern, data in map(None, pattern, data):
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	461 same, vars = match(pattern, data, vars)
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	462 if not same:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	463 break
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	464 return same, vars
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	465 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	466 %
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	467 Using this simple representation for syntactic variables and the symbolic
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	468 node types, the pattern for the candidate docstring subtrees becomes
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	469 fairly readable. (See file \file{example.py}.)
3198 21cd50a4a3f7 Fred Drake's parser module Guido van Rossum <guido@python.org> parents: diff changeset	470
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	471 \begin{verbatim}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	472 import symbol
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	473 import token
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	474
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	475 DOCSTRING_STMT_PATTERN = (
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	476 symbol.stmt,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	477 (symbol.simple_stmt,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	478 (symbol.small_stmt,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	479 (symbol.expr_stmt,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	480 (symbol.testlist,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	481 (symbol.test,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	482 (symbol.and_test,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	483 (symbol.not_test,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	484 (symbol.comparison,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	485 (symbol.expr,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	486 (symbol.xor_expr,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	487 (symbol.and_expr,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	488 (symbol.shift_expr,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	489 (symbol.arith_expr,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	490 (symbol.term,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	491 (symbol.factor,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	492 (symbol.power,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	493 (symbol.atom,
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	494 (token.STRING, ['docstring'])
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	495 )))))))))))))))),
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	496 (token.NEWLINE, '')
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	497 ))
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	498 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	499 %
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	500 Using the \function{match()} function with this pattern, extracting the
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	501 module docstring from the parse tree created previously is easy:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	502
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	503 \begin{verbatim}
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	504 >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1])
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	505 >>> found
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	506 1
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	507 >>> vars
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	508 {'docstring': '"""Some documentation.\012"""'}
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	509 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	510 %
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	511 Once specific data can be extracted from a location where it is
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	512 expected, the question of where information can be expected
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	513 needs to be answered. When dealing with docstrings, the answer is
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	514 fairly simple: the docstring is the first \constant{stmt} node in a code
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	515 block (\constant{file_input} or \constant{suite} node types). A module
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	516 consists of a single \constant{file_input} node, and class and function
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	517 definitions each contain exactly one \constant{suite} node. Classes and
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	518 functions are readily identified as subtrees of code block nodes which
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	519 start with \code{(stmt, (compound_stmt, (classdef, ...} or
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	520 \code{(stmt, (compound_stmt, (funcdef, ...}. Note that these subtrees
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	521 cannot be matched by \function{match()} since it does not support multiple
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	522 sibling nodes to match without regard to number. A more elaborate
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	523 matching function could be used to overcome this limitation, but this
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	524 is sufficient for the example.
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	525
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	526 Given the ability to determine whether a statement might be a
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	527 docstring and extract the actual string from the statement, some work
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	528 needs to be performed to walk the parse tree for an entire module and
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	529 extract information about the names defined in each context of the
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	530 module and associate any docstrings with the names. The code to
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	531 perform this work is not complicated, but bears some explanation.
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	532
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	533 The public interface to the classes is straightforward and should
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	534 probably be somewhat more flexible. Each ``major'' block of the
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	535 module is described by an object providing several methods for inquiry
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	536 and a constructor which accepts at least the subtree of the complete
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	537 parse tree which it represents. The \code{ModuleInfo} constructor
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	538 accepts an optional \code{\var{name}} parameter since it cannot
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	539 otherwise determine the name of the module.
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	540
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	541 The public classes include \class{ClassInfo}, \class{FunctionInfo},
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	542 and \class{ModuleInfo}. All objects provide the
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	543 methods \method{get_name()}, \method{get_docstring()},
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	544 \method{get_class_names()}, and \method{get_class_info()}. The
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	545 \class{ClassInfo} objects support \method{get_method_names()} and
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	546 \method{get_method_info()} while the other classes provide
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	547 \method{get_function_names()} and \method{get_function_info()}.
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	548
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	549 Within each of the forms of code block that the public classes
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	550 represent, most of the required information is in the same form and is
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	551 accessed in the same way, with classes having the distinction that
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	552 functions defined at the top level are referred to as ``methods.''
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	553 Since the difference in nomenclature reflects a real semantic
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	554 distinction from functions defined outside of a class, the
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	555 implementation needs to maintain the distinction.
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	556 Hence, most of the functionality of the public classes can be
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	557 implemented in a common base class, \class{SuiteInfoBase}, with the
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	558 accessors for function and method information provided elsewhere.
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	559 Note that there is only one class which represents function and method
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	560 information; this parallels the use of the \keyword{def} statement to
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	561 define both types of elements.
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	562
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	563 Most of the accessor functions are declared in \class{SuiteInfoBase}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	564 and do not need to be overriden by subclasses. More importantly, the
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	565 extraction of most information from a parse tree is handled through a
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	566 method called by the \class{SuiteInfoBase} constructor. The example
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	567 code for most of the classes is clear when read alongside the formal
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	568 grammar, but the method which recursively creates new information
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	569 objects requires further examination. Here is the relevant part of
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	570 the \class{SuiteInfoBase} definition from \file{example.py}:
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	571
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	572 \begin{verbatim}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	573 class SuiteInfoBase:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	574 _docstring = ''
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	575 _name = ''
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	576
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	577 def __init__(self, tree = None):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	578 self._class_info = {}
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	579 self._function_info = {}
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	580 if tree:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	581 self._extract_info(tree)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	582
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	583 def _extract_info(self, tree):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	584 # extract docstring
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	585 if len(tree) == 2:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	586 found, vars = match(DOCSTRING_STMT_PATTERN[1], tree[1])
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	587 else:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	588 found, vars = match(DOCSTRING_STMT_PATTERN, tree[3])
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	589 if found:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	590 self._docstring = eval(vars['docstring'])
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	591 # discover inner definitions
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	592 for node in tree[1:]:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	593 found, vars = match(COMPOUND_STMT_PATTERN, node)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	594 if found:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	595 cstmt = vars['compound']
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	596 if cstmt[0] == symbol.funcdef:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	597 name = cstmt[2][1]
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	598 self._function_info[name] = FunctionInfo(cstmt)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	599 elif cstmt[0] == symbol.classdef:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	600 name = cstmt[2][1]
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	601 self._class_info[name] = ClassInfo(cstmt)
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	602 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	603 %
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	604 After initializing some internal state, the constructor calls the
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	605 \method{_extract_info()} method. This method performs the bulk of the
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	606 information extraction which takes place in the entire example. The
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	607 extraction has two distinct phases: the location of the docstring for
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	608 the parse tree passed in, and the discovery of additional definitions
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	609 within the code block represented by the parse tree.
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	610
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	611 The initial \keyword{if} test determines whether the nested suite is of
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	612 the ``short form'' or the ``long form.'' The short form is used when
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	613 the code block is on the same line as the definition of the code
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	614 block, as in
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	615
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	616 \begin{verbatim}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	617 def square(x): "Square an argument."; return x ** 2
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	618 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	619 %
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	620 while the long form uses an indented block and allows nested
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	621 definitions:
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	622
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	623 \begin{verbatim}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	624 def make_power(exp):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	625 "Make a function that raises an argument to the exponent `exp'."
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	626 def raiser(x, y=exp):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	627 return x ** y
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	628 return raiser
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	629 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	630 %
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	631 When the short form is used, the code block may contain a docstring as
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	632 the first, and possibly only, \constant{small_stmt} element. The
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	633 extraction of such a docstring is slightly different and requires only
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	634 a portion of the complete pattern used in the more common case. As
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	635 implemented, the docstring will only be found if there is only
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	636 one \constant{small_stmt} node in the \constant{simple_stmt} node.
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	637 Since most functions and methods which use the short form do not
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	638 provide a docstring, this may be considered sufficient. The
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	639 extraction of the docstring proceeds using the \function{match()} function
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	640 as described above, and the value of the docstring is stored as an
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	641 attribute of the \class{SuiteInfoBase} object.
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	642
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	643 After docstring extraction, a simple definition discovery
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	644 algorithm operates on the \constant{stmt} nodes of the
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	645 \constant{suite} node. The special case of the short form is not
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	646 tested; since there are no \constant{stmt} nodes in the short form,
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	647 the algorithm will silently skip the single \constant{simple_stmt}
9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	648 node and correctly not discover any nested definitions.
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	649
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	650 Each statement in the code block is categorized as
db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	651 a class definition, function or method definition, or
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	652 something else. For the definition statements, the name of the
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	653 element defined is extracted and a representation object
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	654 appropriate to the definition is created with the defining subtree
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	655 passed as an argument to the constructor. The repesentation objects
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	656 are stored in instance variables and may be retrieved by name using
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	657 the appropriate accessor methods.
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	658
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	659 The public classes provide any accessors required which are more
7223 9dde07a593d8 Add some index entries; switch to logical markup. Fred Drake <fdrake@acm.org> parents: 7050 diff changeset	660 specific than those provided by the \class{SuiteInfoBase} class, but
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	661 the real extraction algorithm remains common to all forms of code
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	662 blocks. A high-level function can be used to extract the complete set
4286 db35d2b93eba (libparser.tex): Revised parser module documentation; improved logical Fred Drake <fdrake@acm.org> parents: 4132 diff changeset	663 of information from a source file. (See file \file{example.py}.)
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	664
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	665 \begin{verbatim}
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	666 def get_docs(fileName):
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	667 source = open(fileName).read()
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	668 import os
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	669 basename = os.path.basename(os.path.splitext(fileName)[0])
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	670 import parser
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	671 ast = parser.suite(source)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	672 tup = parser.ast2tuple(ast)
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	673 return ModuleInfo(tup, basename)
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	674 \end{verbatim}
5818 3468ea8645ca AMK's megapatch: Guido van Rossum <guido@python.org> parents: 5032 diff changeset	675 %
4132 c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	676 This provides an easy-to-use interface to the documentation of a
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	677 module. If information is required which is not extracted by the code
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	678 of this example, the code may be extended at clearly defined points to
c18d52e06238 Final set of changes by Fred before 1.4beta3 Guido van Rossum <guido@python.org> parents: 4084 diff changeset	679 provide additional capabilities.
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	680
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	681 \begin{seealso}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	682
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	683 \seemodule{symbol}%
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	684 {useful constants representing internal nodes of the parse tree}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	685
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	686 \seemodule{token}%
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	687 {useful constants representing leaf nodes of the parse tree and
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	688 functions for testing node values}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	689
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	690 \end{seealso}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	691
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3759 diff changeset	692
6408 184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	693 \section{Standard Module \sectcode{symbol}}
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	694 \label{module-symbol}
6408 184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	695 \stmodindex{symbol}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	696
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	697 This module provides constants which represent the numeric values of
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	698 internal nodes of the parse tree. Unlike most Python constants, these
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	699 use lower-case names. Refer to the file \file{Grammar/Grammar} in the
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	700 Python distribution for the defintions of the names in the context of
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	701 the language grammar. The specific numeric values which the names map
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	702 to may change between Python versions.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	703
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	704 This module also provides one additional data object:
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	705
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	706 \setindexsubitem{(in module symbol)}
6648 0f2498b847bf Removed leading comment about this being "introductory" documentation; Fred Drake <fdrake@acm.org> parents: 6408 diff changeset	707
0f2498b847bf Removed leading comment about this being "introductory" documentation; Fred Drake <fdrake@acm.org> parents: 6408 diff changeset	708
6408 184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	709 \begin{datadesc}{sym_name}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	710 Dictionary mapping the numeric values of the constants defined in this
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	711 module back to name strings, allowing more human-readable
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	712 representation of parse trees to be generated.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	713 \end{datadesc}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	714
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	715 \begin{seealso}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	716 \seemodule{parser}{second example uses this module}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	717 \end{seealso}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	718
6408 184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	719
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	720 \section{Standard Module \sectcode{token}}
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	721 \label{module-token}
6408 184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	722 \stmodindex{token}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	723
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	724 This module provides constants which represent the numeric values of
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	725 leaf nodes of the parse tree (terminal tokens). Refer to the file
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	726 \file{Grammar/Grammar} in the Python distribution for the defintions
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	727 of the names in the context of the language grammar. The specific
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	728 numeric values which the names map to may change between Python
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	729 versions.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	730
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	731 This module also provides one data object and some functions. The
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	732 functions mirror definitions in the Python C header files.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	733
7277 bf4e790bd61a Remove all \bcode / \ecode cruft; this is no longer needed. See previous Fred Drake <fdrake@acm.org> parents: 7223 diff changeset	734 \setindexsubitem{(in module token)}
6648 0f2498b847bf Removed leading comment about this being "introductory" documentation; Fred Drake <fdrake@acm.org> parents: 6408 diff changeset	735
0f2498b847bf Removed leading comment about this being "introductory" documentation; Fred Drake <fdrake@acm.org> parents: 6408 diff changeset	736
6408 184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	737 \begin{datadesc}{tok_name}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	738 Dictionary mapping the numeric values of the constants defined in this
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	739 module back to name strings, allowing more human-readable
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	740 representation of parse trees to be generated.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	741 \end{datadesc}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	742
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	743 \begin{funcdesc}{ISTERMINAL}{x}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	744 Return true for terminal token values.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	745 \end{funcdesc}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	746
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	747 \begin{funcdesc}{ISNONTERMINAL}{x}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	748 Return true for non-terminal token values.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	749 \end{funcdesc}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	750
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	751 \begin{funcdesc}{ISEOF}{x}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	752 Return true if \var{x} is the marker indicating the end of input.
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	753 \end{funcdesc}
184093d96d2a Minor nits (Lib/token.py is a file, token is a module...). Fred Drake <fdrake@acm.org> parents: 5818 diff changeset	754
7050 90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	755 \begin{seealso}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	756 \seemodule{parser}{second example uses this module}
90edcb38fb00 Added \label{} to allow module references. Fred Drake <fdrake@acm.org> parents: 6648 diff changeset	757 \end{seealso}

Mercurial > cpython-lru_cache-weakref

annotate Doc/libparser.tex @ 7278:59d2761ba1cf legacy-trunk