cpython-lru_cache-weakref: Lib/lib2to3/refactor.py annotate

annotate Lib/lib2to3/refactor.py @ 79675:c0a2294de787 2.7

Issue #16199: Backport 2to3 typo fix from 3.2.

author	Chris Jerdonek <chris.jerdonek@gmail.com>
date	Thu, 11 Oct 2012 16:15:46 -0700
parents	fd9ffd10f1c7
children

rev	line source
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	1 # Copyright 2006 Google, Inc. All Rights Reserved.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	2 # Licensed to PSF under a Contributor Agreement.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	3
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	4 """Refactoring framework.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	5
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	6 Used as a main program, this can refactor any number of files and/or
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	7 recursively descend down directories. Imported as a module, this
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	8 provides infrastructure to write your own refactoring tool.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	9 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	10
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	11 from __future__ import with_statement
e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	12
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	13 __author__ = "Guido van Rossum <guido@python.org>"
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	14
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	15
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	16 # Python imports
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	17 import os
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	18 import sys
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	19 import logging
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	20 import operator
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	21 import collections
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	22 import StringIO
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	23 from itertools import chain
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	24
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	25 # Local imports
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	26 from .pgen2 import driver, tokenize, token
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	27 from .fixer_util import find_root
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	28 from . import pytree, pygram
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	29 from . import btm_utils as bu
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	30 from . import btm_matcher as bm
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	31
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	32
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	33 def get_all_fix_names(fixer_pkg, remove_prefix=True):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	34 """Return a sorted list of all available fix names in the given package."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	35 pkg = __import__(fixer_pkg, [], [], ["*"])
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	36 fixer_dir = os.path.dirname(pkg.__file__)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	37 fix_names = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	38 for name in sorted(os.listdir(fixer_dir)):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	39 if name.startswith("fix_") and name.endswith(".py"):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	40 if remove_prefix:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	41 name = name[4:]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	42 fix_names.append(name[:-3])
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	43 return fix_names
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	44
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	45
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	46 class _EveryNode(Exception):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	47 pass
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	48
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	49
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	50 def _get_head_types(pat):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	51 """ Accepts a pytree Pattern Node and returns a set
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	52 of the pattern types which will match first. """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	53
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	54 if isinstance(pat, (pytree.NodePattern, pytree.LeafPattern)):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	55 # NodePatters must either have no type and no content
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	56 # or a type and content -- so they don't get any farther
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	57 # Always return leafs
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	58 if pat.type is None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	59 raise _EveryNode
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	60 return set([pat.type])
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	61
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	62 if isinstance(pat, pytree.NegatedPattern):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	63 if pat.content:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	64 return _get_head_types(pat.content)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	65 raise _EveryNode # Negated Patterns don't have a type
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	66
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	67 if isinstance(pat, pytree.WildcardPattern):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	68 # Recurse on each node in content
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	69 r = set()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	70 for p in pat.content:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	71 for x in p:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	72 r.update(_get_head_types(x))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	73 return r
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	74
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	75 raise Exception("Oh no! I don't understand pattern %s" %(pat))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	76
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	77
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	78 def _get_headnode_dict(fixer_list):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	79 """ Accepts a list of fixers and returns a dictionary
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	80 of head node type --> fixer list. """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	81 head_nodes = collections.defaultdict(list)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	82 every = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	83 for fixer in fixer_list:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	84 if fixer.pattern:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	85 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	86 heads = _get_head_types(fixer.pattern)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	87 except _EveryNode:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	88 every.append(fixer)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	89 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	90 for node_type in heads:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	91 head_nodes[node_type].append(fixer)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	92 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	93 if fixer._accept_type is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	94 head_nodes[fixer._accept_type].append(fixer)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	95 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	96 every.append(fixer)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	97 for node_type in chain(pygram.python_grammar.symbol2number.itervalues(),
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	98 pygram.python_grammar.tokens):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	99 head_nodes[node_type].extend(every)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	100 return dict(head_nodes)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	101
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	102
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	103 def get_fixers_from_package(pkg_name):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	104 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	105 Return the fully qualified names for fixers in the package pkg_name.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	106 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	107 return [pkg_name + "." + fix_name
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	108 for fix_name in get_all_fix_names(pkg_name, False)]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	109
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	110 def _identity(obj):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	111 return obj
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	112
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	113 if sys.version_info < (3, 0):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	114 import codecs
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	115 _open_with_encoding = codecs.open
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	116 # codecs.open doesn't translate newlines sadly.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	117 def _from_system_newlines(input):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	118 return input.replace(u"\r\n", u"\n")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	119 def _to_system_newlines(input):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	120 if os.linesep != "\n":
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	121 return input.replace(u"\n", os.linesep)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	122 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	123 return input
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	124 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	125 _open_with_encoding = open
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	126 _from_system_newlines = _identity
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	127 _to_system_newlines = _identity
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	128
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	129
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	130 def _detect_future_features(source):
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	131 have_docstring = False
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	132 gen = tokenize.generate_tokens(StringIO.StringIO(source).readline)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	133 def advance():
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	134 tok = gen.next()
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	135 return tok[0], tok[1]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	136 ignore = frozenset((token.NEWLINE, tokenize.NL, token.COMMENT))
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	137 features = set()
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	138 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	139 while True:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	140 tp, value = advance()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	141 if tp in ignore:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	142 continue
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	143 elif tp == token.STRING:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	144 if have_docstring:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	145 break
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	146 have_docstring = True
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	147 elif tp == token.NAME and value == u"from":
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	148 tp, value = advance()
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	149 if tp != token.NAME or value != u"__future__":
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	150 break
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	151 tp, value = advance()
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	152 if tp != token.NAME or value != u"import":
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	153 break
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	154 tp, value = advance()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	155 if tp == token.OP and value == u"(":
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	156 tp, value = advance()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	157 while tp == token.NAME:
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	158 features.add(value)
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	159 tp, value = advance()
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	160 if tp != token.OP or value != u",":
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	161 break
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	162 tp, value = advance()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	163 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	164 break
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	165 except StopIteration:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	166 pass
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	167 return frozenset(features)
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	168
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	169
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	170 class FixerError(Exception):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	171 """A fixer could not be loaded."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	172
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	173
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	174 class RefactoringTool(object):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	175
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	176 _default_options = {"print_function" : False,
9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	177 "write_unchanged_files" : False}
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	178
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	179 CLASS_PREFIX = "Fix" # The prefix for fixer classes
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	180 FILE_PREFIX = "fix_" # The prefix for modules with a fixer within
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	181
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	182 def __init__(self, fixer_names, options=None, explicit=None):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	183 """Initializer.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	184
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	185 Args:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	186 fixer_names: a list of fixers to import
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	187 options: an dict with configuration.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	188 explicit: a list of fixers to run even if they are explicit.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	189 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	190 self.fixers = fixer_names
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	191 self.explicit = explicit or []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	192 self.options = self._default_options.copy()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	193 if options is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	194 self.options.update(options)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	195 if self.options["print_function"]:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	196 self.grammar = pygram.python_grammar_no_print_statement
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	197 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	198 self.grammar = pygram.python_grammar
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	199 # When this is True, the refactor*() methods will call write_file() for
9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	200 # files processed even if they were not changed during refactoring. If
9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	201 # and only if the refactor method's write parameter was True.
9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	202 self.write_unchanged_files = self.options.get("write_unchanged_files")
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	203 self.errors = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	204 self.logger = logging.getLogger("RefactoringTool")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	205 self.fixer_log = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	206 self.wrote = False
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	207 self.driver = driver.Driver(self.grammar,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	208 convert=pytree.convert,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	209 logger=self.logger)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	210 self.pre_order, self.post_order = self.get_fixers()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	211
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	212
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	213 self.files = [] # List of files that were or should be modified
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	214
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	215 self.BM = bm.BottomMatcher()
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	216 self.bmi_pre_order = [] # Bottom Matcher incompatible fixers
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	217 self.bmi_post_order = []
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	218
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	219 for fixer in chain(self.post_order, self.pre_order):
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	220 if fixer.BM_compatible:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	221 self.BM.add_fixer(fixer)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	222 # remove fixers that will be handled by the bottom-up
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	223 # matcher
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	224 elif fixer in self.pre_order:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	225 self.bmi_pre_order.append(fixer)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	226 elif fixer in self.post_order:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	227 self.bmi_post_order.append(fixer)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	228
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	229 self.bmi_pre_order_heads = _get_headnode_dict(self.bmi_pre_order)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	230 self.bmi_post_order_heads = _get_headnode_dict(self.bmi_post_order)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	231
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	232
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	233
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	234 def get_fixers(self):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	235 """Inspects the options to load the requested patterns and handlers.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	236
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	237 Returns:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	238 (pre_order, post_order), where pre_order is the list of fixers that
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	239 want a pre-order AST traversal, and post_order is the list that want
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	240 post-order traversal.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	241 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	242 pre_order_fixers = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	243 post_order_fixers = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	244 for fix_mod_path in self.fixers:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	245 mod = __import__(fix_mod_path, {}, {}, ["*"])
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	246 fix_name = fix_mod_path.rsplit(".", 1)[-1]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	247 if fix_name.startswith(self.FILE_PREFIX):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	248 fix_name = fix_name[len(self.FILE_PREFIX):]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	249 parts = fix_name.split("_")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	250 class_name = self.CLASS_PREFIX + "".join([p.title() for p in parts])
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	251 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	252 fix_class = getattr(mod, class_name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	253 except AttributeError:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	254 raise FixerError("Can't find %s.%s" % (fix_name, class_name))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	255 fixer = fix_class(self.options, self.fixer_log)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	256 if fixer.explicit and self.explicit is not True and \
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	257 fix_mod_path not in self.explicit:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	258 self.log_message("Skipping implicit fixer: %s", fix_name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	259 continue
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	260
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	261 self.log_debug("Adding transformation: %s", fix_name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	262 if fixer.order == "pre":
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	263 pre_order_fixers.append(fixer)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	264 elif fixer.order == "post":
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	265 post_order_fixers.append(fixer)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	266 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	267 raise FixerError("Illegal fixer order: %r" % fixer.order)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	268
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	269 key_func = operator.attrgetter("run_order")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	270 pre_order_fixers.sort(key=key_func)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	271 post_order_fixers.sort(key=key_func)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	272 return (pre_order_fixers, post_order_fixers)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	273
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	274 def log_error(self, msg, args, *kwds):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	275 """Called when an error occurs."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	276 raise
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	277
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	278 def log_message(self, msg, *args):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	279 """Hook to log a message."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	280 if args:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	281 msg = msg % args
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	282 self.logger.info(msg)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	283
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	284 def log_debug(self, msg, *args):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	285 if args:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	286 msg = msg % args
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	287 self.logger.debug(msg)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	288
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	289 def print_output(self, old_text, new_text, filename, equal):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	290 """Called with the old version, new version, and filename of a
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	291 refactored file."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	292 pass
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	293
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	294 def refactor(self, items, write=False, doctests_only=False):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	295 """Refactor a list of files and directories."""
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	296
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	297 for dir_or_file in items:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	298 if os.path.isdir(dir_or_file):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	299 self.refactor_dir(dir_or_file, write, doctests_only)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	300 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	301 self.refactor_file(dir_or_file, write, doctests_only)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	302
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	303 def refactor_dir(self, dir_name, write=False, doctests_only=False):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	304 """Descends down a directory and refactor every Python file found.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	305
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	306 Python files are assumed to have a .py extension.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	307
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	308 Files and subdirectories starting with '.' are skipped.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	309 """
66671 d29b55ac1631 Merged revisions 85551,86156-86157,86464 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 65340 diff changeset	310 py_ext = os.extsep + "py"
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	311 for dirpath, dirnames, filenames in os.walk(dir_name):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	312 self.log_debug("Descending into %s", dirpath)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	313 dirnames.sort()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	314 filenames.sort()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	315 for name in filenames:
66671 d29b55ac1631 Merged revisions 85551,86156-86157,86464 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 65340 diff changeset	316 if (not name.startswith(".") and
d29b55ac1631 Merged revisions 85551,86156-86157,86464 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 65340 diff changeset	317 os.path.splitext(name)[1] == py_ext):
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	318 fullname = os.path.join(dirpath, name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	319 self.refactor_file(fullname, write, doctests_only)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	320 # Modify dirnames in-place to remove subdirs with leading dots
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	321 dirnames[:] = [dn for dn in dirnames if not dn.startswith(".")]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	322
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	323 def _read_python_source(self, filename):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	324 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	325 Do our best to decode a Python source file correctly.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	326 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	327 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	328 f = open(filename, "rb")
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	329 except IOError as err:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	330 self.log_error("Can't open %s: %s", filename, err)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	331 return None, None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	332 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	333 encoding = tokenize.detect_encoding(f.readline)[0]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	334 finally:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	335 f.close()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	336 with _open_with_encoding(filename, "r", encoding=encoding) as f:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	337 return _from_system_newlines(f.read()), encoding
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	338
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	339 def refactor_file(self, filename, write=False, doctests_only=False):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	340 """Refactors a file."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	341 input, encoding = self._read_python_source(filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	342 if input is None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	343 # Reading the file failed.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	344 return
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	345 input += u"\n" # Silence certain parse errors
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	346 if doctests_only:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	347 self.log_debug("Refactoring doctests in %s", filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	348 output = self.refactor_docstring(input, filename)
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	349 if self.write_unchanged_files or output != input:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	350 self.processed_file(output, filename, input, write, encoding)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	351 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	352 self.log_debug("No doctest changes in %s", filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	353 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	354 tree = self.refactor_string(input, filename)
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	355 if self.write_unchanged_files or (tree and tree.was_changed):
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	356 # The [:-1] is to take off the \n we added earlier
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	357 self.processed_file(unicode(tree)[:-1], filename,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	358 write=write, encoding=encoding)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	359 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	360 self.log_debug("No changes in %s", filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	361
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	362 def refactor_string(self, data, name):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	363 """Refactor a given input string.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	364
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	365 Args:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	366 data: a string holding the code to be refactored.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	367 name: a human-readable name for use in error/log messages.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	368
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	369 Returns:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	370 An AST corresponding to the refactored input stream; None if
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	371 there were errors during the parse.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	372 """
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	373 features = _detect_future_features(data)
e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	374 if "print_function" in features:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	375 self.driver.grammar = pygram.python_grammar_no_print_statement
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	376 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	377 tree = self.driver.parse_string(data)
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	378 except Exception as err:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	379 self.log_error("Can't parse %s: %s: %s",
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	380 name, err.__class__.__name__, err)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	381 return
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	382 finally:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	383 self.driver.grammar = self.grammar
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	384 tree.future_features = features
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	385 self.log_debug("Refactoring %s", name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	386 self.refactor_tree(tree, name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	387 return tree
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	388
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	389 def refactor_stdin(self, doctests_only=False):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	390 input = sys.stdin.read()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	391 if doctests_only:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	392 self.log_debug("Refactoring doctests in stdin")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	393 output = self.refactor_docstring(input, "<stdin>")
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	394 if self.write_unchanged_files or output != input:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	395 self.processed_file(output, "<stdin>", input)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	396 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	397 self.log_debug("No doctest changes in stdin")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	398 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	399 tree = self.refactor_string(input, "<stdin>")
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	400 if self.write_unchanged_files or (tree and tree.was_changed):
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	401 self.processed_file(unicode(tree), "<stdin>", input)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	402 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	403 self.log_debug("No changes in stdin")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	404
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	405 def refactor_tree(self, tree, name):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	406 """Refactors a parse tree (modifying the tree in place).
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	407
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	408 For compatible patterns the bottom matcher module is
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	409 used. Otherwise the tree is traversed node-to-node for
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	410 matches.
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	411
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	412 Args:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	413 tree: a pytree.Node instance representing the root of the tree
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	414 to be refactored.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	415 name: a human-readable name for this tree.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	416
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	417 Returns:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	418 True if the tree was modified, False otherwise.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	419 """
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	420
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	421 for fixer in chain(self.pre_order, self.post_order):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	422 fixer.start_tree(tree, name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	423
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	424 #use traditional matching for the incompatible fixers
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	425 self.traverse_by(self.bmi_pre_order_heads, tree.pre_order())
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	426 self.traverse_by(self.bmi_post_order_heads, tree.post_order())
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	427
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	428 # obtain a set of candidate nodes
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	429 match_set = self.BM.run(tree.leaves())
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	430
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	431 while any(match_set.values()):
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	432 for fixer in self.BM.fixers:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	433 if fixer in match_set and match_set[fixer]:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	434 #sort by depth; apply fixers from bottom(of the AST) to top
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	435 match_set[fixer].sort(key=pytree.Base.depth, reverse=True)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	436
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	437 if fixer.keep_line_order:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	438 #some fixers(eg fix_imports) must be applied
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	439 #with the original file's line order
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	440 match_set[fixer].sort(key=pytree.Base.get_lineno)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	441
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	442 for node in list(match_set[fixer]):
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	443 if node in match_set[fixer]:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	444 match_set[fixer].remove(node)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	445
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	446 try:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	447 find_root(node)
79178 fd9ffd10f1c7 switch assertion to an explicit ValueError Benjamin Peterson <benjamin@python.org> parents: 74896 diff changeset	448 except ValueError:
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	449 # this node has been cut off from a
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	450 # previous transformation ; skip
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	451 continue
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	452
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	453 if node.fixers_applied and fixer in node.fixers_applied:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	454 # do not apply the same fixer again
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	455 continue
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	456
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	457 results = fixer.match(node)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	458
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	459 if results:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	460 new = fixer.transform(node, results)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	461 if new is not None:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	462 node.replace(new)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	463 #new.fixers_applied.append(fixer)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	464 for node in new.post_order():
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	465 # do not apply the fixer again to
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	466 # this or any subnode
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	467 if not node.fixers_applied:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	468 node.fixers_applied = []
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	469 node.fixers_applied.append(fixer)
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	470
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	471 # update the original match set for
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	472 # the added code
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	473 new_matches = self.BM.run(new.leaves())
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	474 for fxr in new_matches:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	475 if not fxr in match_set:
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	476 match_set[fxr]=[]
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	477
7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	478 match_set[fxr].extend(new_matches[fxr])
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	479
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	480 for fixer in chain(self.pre_order, self.post_order):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	481 fixer.finish_tree(tree, name)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	482 return tree.was_changed
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	483
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	484 def traverse_by(self, fixers, traversal):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	485 """Traverse an AST, applying a set of fixers to each node.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	486
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	487 This is a helper method for refactor_tree().
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	488
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	489 Args:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	490 fixers: a list of fixer instances.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	491 traversal: a generator that yields AST nodes.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	492
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	493 Returns:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	494 None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	495 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	496 if not fixers:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	497 return
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	498 for node in traversal:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	499 for fixer in fixers[node.type]:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	500 results = fixer.match(node)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	501 if results:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	502 new = fixer.transform(node, results)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	503 if new is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	504 node.replace(new)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	505 node = new
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	506
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	507 def processed_file(self, new_text, filename, old_text=None, write=False,
68996 4f07cacb2c3b revert unintended changes Benjamin Peterson <benjamin@python.org> parents: 68992 diff changeset	508 encoding=None):
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	509 """
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	510 Called when a file has been refactored and there may be changes.
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	511 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	512 self.files.append(filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	513 if old_text is None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	514 old_text = self._read_python_source(filename)[0]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	515 if old_text is None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	516 return
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	517 equal = old_text == new_text
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	518 self.print_output(old_text, new_text, filename, equal)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	519 if equal:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	520 self.log_debug("No changes to %s", filename)
74896 9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	521 if not self.write_unchanged_files:
9f583700d27f Issue #13930: Adds ability for 2to3 to write its output to a different Gregory P. Smith <greg@krypto.org> parents: 68996 diff changeset	522 return
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	523 if write:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	524 self.write_file(new_text, filename, old_text, encoding)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	525 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	526 self.log_debug("Not writing changes to %s", filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	527
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	528 def write_file(self, new_text, filename, old_text, encoding=None):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	529 """Writes a string to a file.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	530
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	531 It first shows a unified diff between the old text and the new text, and
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	532 then rewrites the file; the latter is only done if the write option is
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	533 set.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	534 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	535 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	536 f = _open_with_encoding(filename, "w", encoding=encoding)
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	537 except os.error as err:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	538 self.log_error("Can't create %s: %s", filename, err)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	539 return
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	540 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	541 f.write(_to_system_newlines(new_text))
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	542 except os.error as err:
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	543 self.log_error("Can't write %s: %s", filename, err)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	544 finally:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	545 f.close()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	546 self.log_debug("Wrote changes to %s", filename)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	547 self.wrote = True
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	548
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	549 PS1 = ">>> "
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	550 PS2 = "... "
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	551
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	552 def refactor_docstring(self, input, filename):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	553 """Refactors a docstring, looking for doctests.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	554
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	555 This returns a modified version of the input string. It looks
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	556 for doctests, which start with a ">>>" prompt, and may be
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	557 continued with "..." prompts, as long as the "..." is indented
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	558 the same as the ">>>".
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	559
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	560 (Unfortunately we can't use the doctest module's parser,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	561 since, like most parsers, it is not geared towards preserving
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	562 the original source.)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	563 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	564 result = []
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	565 block = None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	566 block_lineno = None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	567 indent = None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	568 lineno = 0
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	569 for line in input.splitlines(True):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	570 lineno += 1
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	571 if line.lstrip().startswith(self.PS1):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	572 if block is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	573 result.extend(self.refactor_doctest(block, block_lineno,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	574 indent, filename))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	575 block_lineno = lineno
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	576 block = [line]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	577 i = line.find(self.PS1)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	578 indent = line[:i]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	579 elif (indent is not None and
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	580 (line.startswith(indent + self.PS2) or
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	581 line == indent + self.PS2.rstrip() + u"\n")):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	582 block.append(line)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	583 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	584 if block is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	585 result.extend(self.refactor_doctest(block, block_lineno,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	586 indent, filename))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	587 block = None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	588 indent = None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	589 result.append(line)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	590 if block is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	591 result.extend(self.refactor_doctest(block, block_lineno,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	592 indent, filename))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	593 return u"".join(result)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	594
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	595 def refactor_doctest(self, block, lineno, indent, filename):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	596 """Refactors one doctest.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	597
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	598 A doctest is given as a block of lines, the first of which starts
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	599 with ">>>" (possibly indented), while the remaining lines start
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	600 with "..." (identically indented).
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	601
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	602 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	603 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	604 tree = self.parse_block(block, lineno, indent)
65340 7d1e453b120b Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 63841 diff changeset	605 except Exception as err:
63841 feffc1a2599c Merged revisions 82530-82531,82779,82855,83740,83789-83791,83797-83801,83803,83811,83827,83844 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 61866 diff changeset	606 if self.logger.isEnabledFor(logging.DEBUG):
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	607 for line in block:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	608 self.log_debug("Source: %s", line.rstrip(u"\n"))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	609 self.log_error("Can't parse docstring in %s line %s: %s: %s",
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	610 filename, lineno, err.__class__.__name__, err)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	611 return block
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	612 if self.refactor_tree(tree, filename):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	613 new = unicode(tree).splitlines(True)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	614 # Undo the adjustment of the line numbers in wrap_toks() below.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	615 clipped, new = new[:lineno-1], new[lineno-1:]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	616 assert clipped == [u"\n"] * (lineno-1), clipped
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	617 if not new[-1].endswith(u"\n"):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	618 new[-1] += u"\n"
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	619 block = [indent + self.PS1 + new.pop(0)]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	620 if new:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	621 block += [indent + self.PS2 + line for line in new]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	622 return block
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	623
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	624 def summarize(self):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	625 if self.wrote:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	626 were = "were"
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	627 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	628 were = "need to be"
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	629 if not self.files:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	630 self.log_message("No files %s modified.", were)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	631 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	632 self.log_message("Files that %s modified:", were)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	633 for file in self.files:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	634 self.log_message(file)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	635 if self.fixer_log:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	636 self.log_message("Warnings/messages while refactoring:")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	637 for message in self.fixer_log:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	638 self.log_message(message)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	639 if self.errors:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	640 if len(self.errors) == 1:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	641 self.log_message("There was 1 error:")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	642 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	643 self.log_message("There were %d errors:", len(self.errors))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	644 for msg, args, kwds in self.errors:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	645 self.log_message(msg, args, *kwds)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	646
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	647 def parse_block(self, block, lineno, indent):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	648 """Parses a block into a tree.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	649
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	650 This is necessary to get correct line number / offset information
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	651 in the parser diagnostics and embedded into the parse tree.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	652 """
61869 4e4952636113 Merged revisions 81479 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 61376 diff changeset	653 tree = self.driver.parse_tokens(self.wrap_toks(block, lineno, indent))
4e4952636113 Merged revisions 81479 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 61376 diff changeset	654 tree.future_features = frozenset()
4e4952636113 Merged revisions 81479 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 61376 diff changeset	655 return tree
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	656
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	657 def wrap_toks(self, block, lineno, indent):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	658 """Wraps a tokenize stream to systematically modify start/end."""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	659 tokens = tokenize.generate_tokens(self.gen_lines(block, indent).next)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	660 for type, value, (line0, col0), (line1, col1), line_text in tokens:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	661 line0 += lineno - 1
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	662 line1 += lineno - 1
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	663 # Don't bother updating the columns; this is too complicated
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	664 # since line_text would also have to be updated and it would
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	665 # still break for tokens spanning lines. Let the user guess
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	666 # that the column numbers for doctests are relative to the
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	667 # end of the prompt string (PS1 or PS2).
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	668 yield type, value, (line0, col0), (line1, col1), line_text
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	669
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	670
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	671 def gen_lines(self, block, indent):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	672 """Generates lines as expected by tokenize from a list of lines.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	673
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	674 This strips the first len(indent + self.PS1) characters off each line.
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	675 """
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	676 prefix1 = indent + self.PS1
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	677 prefix2 = indent + self.PS2
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	678 prefix = prefix1
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	679 for line in block:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	680 if line.startswith(prefix):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	681 yield line[len(prefix):]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	682 elif line == prefix.rstrip() + u"\n":
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	683 yield u"\n"
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	684 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	685 raise AssertionError("line=%r, prefix=%r" % (line, prefix))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	686 prefix = prefix2
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	687 while True:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	688 yield ""
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	689
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	690
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	691 class MultiprocessingUnsupported(Exception):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	692 pass
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	693
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	694
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	695 class MultiprocessRefactoringTool(RefactoringTool):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	696
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	697 def __init__(self, args, *kwargs):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	698 super(MultiprocessRefactoringTool, self).__init__(args, *kwargs)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	699 self.queue = None
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	700 self.output_lock = None
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	701
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	702 def refactor(self, items, write=False, doctests_only=False,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	703 num_processes=1):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	704 if num_processes == 1:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	705 return super(MultiprocessRefactoringTool, self).refactor(
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	706 items, write, doctests_only)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	707 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	708 import multiprocessing
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	709 except ImportError:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	710 raise MultiprocessingUnsupported
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	711 if self.queue is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	712 raise RuntimeError("already doing multiple processes")
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	713 self.queue = multiprocessing.JoinableQueue()
61376 e92fb95f2317 Merged revisions 80934 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 58737 diff changeset	714 self.output_lock = multiprocessing.Lock()
58737 fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	715 processes = [multiprocessing.Process(target=self._child)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	716 for i in xrange(num_processes)]
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	717 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	718 for p in processes:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	719 p.start()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	720 super(MultiprocessRefactoringTool, self).refactor(items, write,
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	721 doctests_only)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	722 finally:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	723 self.queue.join()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	724 for i in xrange(num_processes):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	725 self.queue.put(None)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	726 for p in processes:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	727 if p.is_alive():
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	728 p.join()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	729 self.queue = None
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	730
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	731 def _child(self):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	732 task = self.queue.get()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	733 while task is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	734 args, kwargs = task
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	735 try:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	736 super(MultiprocessRefactoringTool, self).refactor_file(
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	737 args, *kwargs)
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	738 finally:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	739 self.queue.task_done()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	740 task = self.queue.get()
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	741
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	742 def refactor_file(self, args, *kwargs):
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	743 if self.queue is not None:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	744 self.queue.put((args, kwargs))
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	745 else:
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	746 return super(MultiprocessRefactoringTool, self).refactor_file(
fab380674959 copy lib2to3 from the trunk Benjamin Peterson <benjamin@python.org> parents: diff changeset	747 args, *kwargs)

Mercurial > cpython-lru_cache-weakref

annotate Lib/lib2to3/refactor.py @ 79675:c0a2294de787 2.7