cpython/Lib/lib2to3/refactor.py
Christian Heimes 81ee3efede Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r62425 | andrew.kuchling | 2008-04-21 03:45:57 +0200 (Mon, 21 Apr 2008) | 1 line

  Comment typo
................
  r62426 | mark.dickinson | 2008-04-21 03:55:50 +0200 (Mon, 21 Apr 2008) | 2 lines

  Silence 'r may be used uninitialized' compiler warning.
................
  r62427 | andrew.kuchling | 2008-04-21 04:08:00 +0200 (Mon, 21 Apr 2008) | 1 line

  Markup fix
................
  r62428 | andrew.kuchling | 2008-04-21 04:08:13 +0200 (Mon, 21 Apr 2008) | 1 line

  Wording changes
................
  r62429 | andrew.kuchling | 2008-04-21 04:14:24 +0200 (Mon, 21 Apr 2008) | 1 line

  Add various items
................
  r62434 | thomas.heller | 2008-04-21 15:46:55 +0200 (Mon, 21 Apr 2008) | 1 line

  Fix typo.
................
  r62435 | david.goodger | 2008-04-21 16:40:22 +0200 (Mon, 21 Apr 2008) | 1 line

  corrections ("reStructuredText" is one word)
................
  r62436 | david.goodger | 2008-04-21 16:43:33 +0200 (Mon, 21 Apr 2008) | 1 line

  capitalization
................
  r62441 | gregory.p.smith | 2008-04-21 19:46:40 +0200 (Mon, 21 Apr 2008) | 2 lines

  explicitly flush after the ... since there wasn't a newline
................
  r62444 | jeroen.ruigrok | 2008-04-21 22:15:39 +0200 (Mon, 21 Apr 2008) | 2 lines

  Windows x64 also falls under VER_PLATFORM_WIN32_NT.
................
  r62446 | gregory.p.smith | 2008-04-21 23:31:08 +0200 (Mon, 21 Apr 2008) | 3 lines

  If sys.stdin is not a tty, fall back to default_getpass after printing
  a warning instead of failing with a termios.error.
................
  r62447 | mark.dickinson | 2008-04-22 00:32:24 +0200 (Tue, 22 Apr 2008) | 8 lines

  test_math and test_cmath are failing on the FreeBSD 6.2 trunk buildbot,
  apparently because tanh(-0.) loses the sign of zero on that platform.
  If true, this is a bug in FreeBSD.

  Added a configure test to verify this.  I still need to figure out
  how best to deal with this failure.
................
  r62448 | amaury.forgeotdarc | 2008-04-22 00:35:30 +0200 (Tue, 22 Apr 2008) | 7 lines

  Issue 2665: On Windows, sys.stderr does not contain a valid file when running without a console.
  It seems to work, but will fail at the first flush.

  This causes IDLE to crash when too many warnings are printed.

  Will backport.
................
  r62450 | benjamin.peterson | 2008-04-22 00:57:00 +0200 (Tue, 22 Apr 2008) | 2 lines

  Fix Sphinx warnings
................
  r62451 | mark.dickinson | 2008-04-22 02:54:27 +0200 (Tue, 22 Apr 2008) | 3 lines

  Make configure test for tanh(-0.) == -0. committed in r62447 actually
  work.  (The test wasn't properly linked with libm.  Sigh.)
................
  r62452 | benjamin.peterson | 2008-04-22 04:16:03 +0200 (Tue, 22 Apr 2008) | 2 lines

  Various io doc updates
................
  r62453 | neal.norwitz | 2008-04-22 07:07:47 +0200 (Tue, 22 Apr 2008) | 1 line

  Add Thomas Lee
................
  r62454 | gregory.p.smith | 2008-04-22 10:08:41 +0200 (Tue, 22 Apr 2008) | 8 lines

  Major improvements:
  * Default to using /dev/tty for the password prompt and input before
    falling back to sys.stdin and sys.stderr.
  * Use sys.stderr instead of sys.stdout.
  * print the 'password may be echoed' warning to stream used to display
    the prompt rather than always sys.stderr.
  * warn() with GetPassWarning when input may be echoed.
................
  r62455 | gregory.p.smith | 2008-04-22 10:11:33 +0200 (Tue, 22 Apr 2008) | 2 lines

  update the getpass entry
................
  r62463 | amaury.forgeotdarc | 2008-04-22 23:14:41 +0200 (Tue, 22 Apr 2008) | 5 lines

  Issue #2670: urllib2.build_opener() failed when two handlers
  derive the same default base class.

  Will backport.
................
  r62465 | skip.montanaro | 2008-04-23 00:45:09 +0200 (Wed, 23 Apr 2008) | 3 lines

  Factor in documentation changes from issue 1753732.
................
  r62466 | gregory.p.smith | 2008-04-23 03:06:42 +0200 (Wed, 23 Apr 2008) | 2 lines

  syntax fixup
................
  r62469 | benjamin.peterson | 2008-04-23 22:38:06 +0200 (Wed, 23 Apr 2008) | 2 lines

  #2673 Fix example typo in optparse docs
................
  r62474 | martin.v.loewis | 2008-04-24 11:50:50 +0200 (Thu, 24 Apr 2008) | 2 lines

  Add Guilherme Polo.
................
  r62476 | martin.v.loewis | 2008-04-24 15:16:36 +0200 (Thu, 24 Apr 2008) | 3 lines

  Remove Py_Refcnt, Py_Type, Py_Size, as they were added only
  for backwards compatibility, yet 2.5 did not have them at all.
................
  r62477 | martin.v.loewis | 2008-04-24 15:17:24 +0200 (Thu, 24 Apr 2008) | 2 lines

  Fix typo.
................
  r62478 | martin.v.loewis | 2008-04-24 15:18:03 +0200 (Thu, 24 Apr 2008) | 2 lines

  Add Jesus Cea.
................
  r62480 | amaury.forgeotdarc | 2008-04-24 20:07:05 +0200 (Thu, 24 Apr 2008) | 4 lines

  Issue2681: the literal 0o8 was wrongly accepted, and evaluated as float(0.0).
  This happened only when 8 is the first digit.
  Credits go to Lukas Meuser.
................
  r62485 | amaury.forgeotdarc | 2008-04-24 22:10:26 +0200 (Thu, 24 Apr 2008) | 5 lines

  Disable gc when running test_trace, or we may record the __del__ of collected objects.

  See http://mail.python.org/pipermail/python-checkins/2008-April/068633.html
  the extra events perfectly match several calls to socket._fileobject.__del__()
................
  r62492 | neal.norwitz | 2008-04-25 05:40:17 +0200 (Fri, 25 Apr 2008) | 1 line

  Fix typo (now -> no)
................
  r62497 | armin.rigo | 2008-04-25 11:35:18 +0200 (Fri, 25 Apr 2008) | 2 lines

  A new crasher.
................
  r62498 | thomas.heller | 2008-04-25 17:44:16 +0200 (Fri, 25 Apr 2008) | 1 line

  Add from_buffer and from_buffer_copy class methods to ctypes types.
................
  r62500 | mark.dickinson | 2008-04-25 18:59:09 +0200 (Fri, 25 Apr 2008) | 3 lines

  Issue 2635: fix bug in the fix_sentence_endings option to textwrap.fill.
................
  r62507 | benjamin.peterson | 2008-04-25 23:43:56 +0200 (Fri, 25 Apr 2008) | 2 lines

  Allow test_import to work when it is invoked directly
................
  r62513 | georg.brandl | 2008-04-26 20:31:07 +0200 (Sat, 26 Apr 2008) | 2 lines

  #2691: document PyLong (s)size_t APIs, patch by Alexander Belopolsky.
................
  r62514 | georg.brandl | 2008-04-26 20:32:17 +0200 (Sat, 26 Apr 2008) | 2 lines

  Add missing return type to dealloc.
................
  r62516 | alexandre.vassalotti | 2008-04-27 02:52:24 +0200 (Sun, 27 Apr 2008) | 2 lines

  Fixed URL of PEP 205 in weakref's module docstring.
................
  r62521 | georg.brandl | 2008-04-27 11:39:59 +0200 (Sun, 27 Apr 2008) | 2 lines

  #2677: add note that not all functions may accept keyword args.
................
  r62531 | georg.brandl | 2008-04-27 19:38:55 +0200 (Sun, 27 Apr 2008) | 2 lines

  Use correct XHTML tags.
................
  r62535 | benjamin.peterson | 2008-04-27 20:14:39 +0200 (Sun, 27 Apr 2008) | 2 lines

  #2700 Document PyNumber_ToBase
................
  r62545 | skip.montanaro | 2008-04-27 22:53:57 +0200 (Sun, 27 Apr 2008) | 1 line

  minor wording changes, rewrap a few lines
................
  r62546 | kurt.kaiser | 2008-04-27 23:07:41 +0200 (Sun, 27 Apr 2008) | 7 lines

  Home / Control-A toggles between left margin and end of leading white
  space.  Patch 1196903 Jeff Shute.

  M    idlelib/PyShell.py
  M    idlelib/EditorWindow.py
  M    idlelib/NEWS.txt
................
  r62548 | kurt.kaiser | 2008-04-27 23:38:05 +0200 (Sun, 27 Apr 2008) | 2 lines

  Improved AutoCompleteWindow logic.  Patch 2062 Tal Einat.
................
  r62549 | kurt.kaiser | 2008-04-27 23:52:19 +0200 (Sun, 27 Apr 2008) | 4 lines

  Autocompletion of filenames now support alternate separators, e.g. the
  '/' char on Windows.  Patch 2061 Tal Einat.
................
  r62550 | skip.montanaro | 2008-04-28 00:49:56 +0200 (Mon, 28 Apr 2008) | 6 lines

  A few small changes:
  * The only exception we should catch when trying to import cStringIO is an
    ImportError.
  * Delete the function signatures embedded in the mk*temp docstrings.
  * The tempdir global variable was initialized twice.
................
  r62551 | skip.montanaro | 2008-04-28 00:52:02 +0200 (Mon, 28 Apr 2008) | 4 lines

  Wrap some long paragraphs and include the default values for optional
  function parameters.
................
  r62553 | skip.montanaro | 2008-04-28 04:57:23 +0200 (Mon, 28 Apr 2008) | 7 lines

  Minor cleanups:
  * Avoid creating unused local variables where we can.  Where we can't prefix
    the unused variables with '_'.
  * Avoid shadowing builtins where it won't change the external interface of a
    function.
  * Use None as default path arg to readmodule and readmodule_ex.
................
  r62554 | skip.montanaro | 2008-04-28 04:59:45 +0200 (Mon, 28 Apr 2008) | 6 lines

  Correct documentation to match implementation: "Class" instead of
  "class_descriptor", "Function" instead of "function_descriptor".  Note
  default path value for readmodule*.  Wrap some long paragraphs.  Don't
  mention 'inpackage' which isn't part of the public API.
................
  r62555 | brett.cannon | 2008-04-28 05:23:50 +0200 (Mon, 28 Apr 2008) | 5 lines

  Fix a bug introduced by the warnings rewrite where tracebacks were being
  improperly indented.

  Closes issue #2699.
................
  r62556 | skip.montanaro | 2008-04-28 05:25:37 +0200 (Mon, 28 Apr 2008) | 2 lines

  Wrap some long lines.
................
  r62557 | skip.montanaro | 2008-04-28 05:27:53 +0200 (Mon, 28 Apr 2008) | 6 lines

  Get rid of _test(), _main(), _debug() and _check().  Tests are no longer
  needed (better set available in Lib/test/test_robotparser.py).  Clean up a
  few PEP 8 nits (compound statements on a single line, whitespace around
  operators).
................
  r62558 | brett.cannon | 2008-04-28 06:50:06 +0200 (Mon, 28 Apr 2008) | 3 lines

  Rename the test_traceback_print() function to traceback_print() to prevent
  test_capi from automatically calling the function.
................
  r62559 | georg.brandl | 2008-04-28 07:16:30 +0200 (Mon, 28 Apr 2008) | 2 lines

  Fix markup.
................
  r62569 | amaury.forgeotdarc | 2008-04-28 23:07:06 +0200 (Mon, 28 Apr 2008) | 5 lines

  test_sundry performs minimal tests (a simple import...) on modules that are not tested otherwise.

  Some of them now have tests and can be removed.
  Only 70 to go...
................
  r62574 | andrew.kuchling | 2008-04-29 04:03:54 +0200 (Tue, 29 Apr 2008) | 1 line

  Strip down SSL docs; I'm not managing to get test programs working, so I'll just give a minimal description
................
  r62577 | martin.v.loewis | 2008-04-29 08:10:53 +0200 (Tue, 29 Apr 2008) | 2 lines

  Add Rodrigo and Heiko.
................
  r62593 | nick.coghlan | 2008-04-30 16:23:36 +0200 (Wed, 30 Apr 2008) | 1 line

  Update command line usage documentation to reflect 2.6 changes (also includes some minor cleanups). Addresses TODO list issue 2258
................
  r62595 | andrew.kuchling | 2008-04-30 18:19:55 +0200 (Wed, 30 Apr 2008) | 1 line

  Typo fix
................
  r62604 | benjamin.peterson | 2008-04-30 23:03:58 +0200 (Wed, 30 Apr 2008) | 2 lines

  make test_support's captured_output a bit more robust when exceptions happen
................
  r62605 | georg.brandl | 2008-04-30 23:08:42 +0200 (Wed, 30 Apr 2008) | 2 lines

  #1748: use functools.wraps instead of rolling own metadata update.
................
  r62606 | benjamin.peterson | 2008-04-30 23:25:55 +0200 (Wed, 30 Apr 2008) | 2 lines

  Remove some from __future__ import with_statements
................
  r62608 | benjamin.peterson | 2008-05-01 00:03:36 +0200 (Thu, 01 May 2008) | 2 lines

  Fix typo in whatsnew
................
  r62616 | georg.brandl | 2008-05-01 20:24:32 +0200 (Thu, 01 May 2008) | 2 lines

  Fix synopsis.
................
  r62626 | brett.cannon | 2008-05-02 04:25:09 +0200 (Fri, 02 May 2008) | 6 lines

  Fix a backwards-compatibility mistake where a new optional argument for
  warnings.showwarning() was being used. This broke pre-existing replacements for
  the function since they didn't support the extra argument.

  Closes issue 2705.
................
  r62627 | gregory.p.smith | 2008-05-02 09:26:52 +0200 (Fri, 02 May 2008) | 20 lines

  This should fix issue2632.  A long description of the two competing
  problems is in the bug report (one old, one recently introduced trying
  to fix the old one).  In short:

  buffer data during socket._fileobject.read() and readlines() within a
  cStringIO object instead of a [] of str()s returned from the recv()
  call.

  This prevents excessive memory use due to the size parameter being
  passed to recv() being grossly larger than the actual size of the data
  returned *and* prevents excessive cpu usage due to looping in python
  calling recv() with a very tiny size value if min() is used as the
  previous memory-use bug "fix" did.

  It also documents what the socket._fileobject._rbufsize member is
  actually used for.

  This is a candidate for back porting to 2.5.
................
  r62636 | mark.hammond | 2008-05-02 14:48:15 +0200 (Fri, 02 May 2008) | 2 lines

  #2581: Vista UAC/elevation support for bdist_wininst
................
  r62638 | facundo.batista | 2008-05-02 19:39:00 +0200 (Fri, 02 May 2008) | 3 lines


  Fixed some test structures. Thanks Mark Dickinson.
................
  r62644 | ronald.oussoren | 2008-05-02 21:45:11 +0200 (Fri, 02 May 2008) | 7 lines

  Fix for issue #2573: Can't change the framework name on OS X builds

  This introduces a new configure option: --with-framework-name=NAME
  (defaulting to 'Python'). This allows you to install several copies
  of the Python framework with different names (such as a normal build
  and a debug build).
................
  r62645 | ronald.oussoren | 2008-05-02 21:58:56 +0200 (Fri, 02 May 2008) | 2 lines

  Finish fix for issue2573, previous patch was incomplete.
................
  r62647 | martin.v.loewis | 2008-05-02 23:30:20 +0200 (Fri, 02 May 2008) | 13 lines

  Merged revisions 62263-62646 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r62470 | david.wolever | 2008-04-24 02:11:07 +0200 (Do, 24 Apr 2008) | 3 lines

    Fixed up and applied the patch for #2431 -- speeding up 2to3 with a lookup table.
  ........
    r62646 | martin.v.loewis | 2008-05-02 23:29:27 +0200 (Fr, 02 Mai 2008) | 2 lines

    Fix whitespace.
  ........
................
  r62648 | ronald.oussoren | 2008-05-02 23:42:35 +0200 (Fri, 02 May 2008) | 4 lines

  Fix for #1905: PythonLauncher not working correctly on OSX 10.5/Leopard

  This fixes both Python Launchar and the terminalcommand module.
................
  r62651 | ronald.oussoren | 2008-05-02 23:54:56 +0200 (Fri, 02 May 2008) | 2 lines

  Fix for issue #2520 (cannot import macerrors)
................
  r62652 | benjamin.peterson | 2008-05-03 00:12:58 +0200 (Sat, 03 May 2008) | 2 lines

  capitalization nit for reStructuredText
................
  r62653 | brett.cannon | 2008-05-03 03:02:41 +0200 (Sat, 03 May 2008) | 2 lines

  Fix some indentation errors.
................
  r62656 | brett.cannon | 2008-05-03 05:19:39 +0200 (Sat, 03 May 2008) | 6 lines

  Fix the C implementation of 'warnings' to infer the filename of the module that
  raised an exception properly when __file__ is not set, __name__ == '__main__',
  and sys.argv[0] is a false value.

  Closes issue2743.
................
  r62661 | amaury.forgeotdarc | 2008-05-03 14:21:13 +0200 (Sat, 03 May 2008) | 8 lines

  In test_io, StatefulIncrementalDecoderTest was not part of the test suite.
  And of course, the test failed:
  a bytearray was used without reason in io.TextIOWrapper.tell().

  The difference is that iterating over bytes (i.e. str in python2.6) returns 1-char bytes,
  whereas bytearrays yield integers.
  This code should still work with python3.0
................
  r62663 | benjamin.peterson | 2008-05-03 17:56:42 +0200 (Sat, 03 May 2008) | 2 lines

  The compiling struct is now passed around to all AST helpers (see issue 2720)
................
  r62680 | benjamin.peterson | 2008-05-03 23:35:18 +0200 (Sat, 03 May 2008) | 2 lines

  Moved testing of builtin types out of test_builtin and into type specific modules
................
  r62686 | mark.dickinson | 2008-05-04 04:25:46 +0200 (Sun, 04 May 2008) | 4 lines

  Make sure that Context traps and flags dictionaries have values 0 and 1
  (as documented) rather than True and False.
................
  r62687 | benjamin.peterson | 2008-05-04 05:05:49 +0200 (Sun, 04 May 2008) | 2 lines

  Fix typo in whatsnew
................
  r62696 | georg.brandl | 2008-05-04 11:15:04 +0200 (Sun, 04 May 2008) | 2 lines

  #2752: wrong meaning of '' for socket host.
................
  r62699 | christian.heimes | 2008-05-04 13:50:53 +0200 (Sun, 04 May 2008) | 1 line

  Added note that Python requires at least Win2k SP4
................
  r62700 | gerhard.haering | 2008-05-04 14:59:57 +0200 (Sun, 04 May 2008) | 3 lines

  SQLite requires 64-bit integers in order to build. So the whole HAVE_LONG_LONG
  #ifdefing was useless.
................
  r62701 | gerhard.haering | 2008-05-04 15:15:12 +0200 (Sun, 04 May 2008) | 3 lines

  Applied sqliterow-richcmp.diff patch from Thomas Heller in Issue2152. The
  sqlite3.Row type is now correctly hashable.
................
  r62702 | gerhard.haering | 2008-05-04 15:42:44 +0200 (Sun, 04 May 2008) | 5 lines

  Implemented feature request 2157: Converter names are cut off at '('
  characters. This avoids the common case of something like 'NUMBER(10)' not
  being parsed as 'NUMBER', like expected. Also corrected the docs about
  converter names being case-sensitive. They aren't any longer.
................
  r62703 | georg.brandl | 2008-05-04 17:45:05 +0200 (Sun, 04 May 2008) | 2 lines

  #2757: Remove spare newline.
................
  r62711 | benjamin.peterson | 2008-05-04 21:10:02 +0200 (Sun, 04 May 2008) | 2 lines

  Fix typo in bugs.rst
................
2008-05-04 22:42:01 +00:00

577 lines
21 KiB
Python
Executable File

#!/usr/bin/env python2.5
# Copyright 2006 Google, Inc. All Rights Reserved.
# Licensed to PSF under a Contributor Agreement.
"""Refactoring framework.
Used as a main program, this can refactor any number of files and/or
recursively descend down directories. Imported as a module, this
provides infrastructure to write your own refactoring tool.
"""
__author__ = "Guido van Rossum <guido@python.org>"
# Python imports
import os
import sys
import difflib
import optparse
import logging
from collections import defaultdict
from itertools import chain
# Local imports
from .pgen2 import driver
from .pgen2 import tokenize
from . import pytree
from . import patcomp
from . import fixes
from . import pygram
def main(args=None):
"""Main program.
Call without arguments to use sys.argv[1:] as the arguments; or
call with a list of arguments (excluding sys.argv[0]).
Returns a suggested exit status (0, 1, 2).
"""
# Set up option parser
parser = optparse.OptionParser(usage="refactor.py [options] file|dir ...")
parser.add_option("-d", "--doctests_only", action="store_true",
help="Fix up doctests only")
parser.add_option("-f", "--fix", action="append", default=[],
help="Each FIX specifies a transformation; default all")
parser.add_option("-l", "--list-fixes", action="store_true",
help="List available transformations (fixes/fix_*.py)")
parser.add_option("-p", "--print-function", action="store_true",
help="Modify the grammar so that print() is a function")
parser.add_option("-v", "--verbose", action="store_true",
help="More verbose logging")
parser.add_option("-w", "--write", action="store_true",
help="Write back modified files")
# Parse command line arguments
options, args = parser.parse_args(args)
if options.list_fixes:
print("Available transformations for the -f/--fix option:")
for fixname in get_all_fix_names():
print(fixname)
if not args:
return 0
if not args:
print("At least one file or directory argument required.", file=sys.stderr)
print("Use --help to show usage.", file=sys.stderr)
return 2
# Set up logging handler
if sys.version_info < (2, 4):
hdlr = logging.StreamHandler()
fmt = logging.Formatter('%(name)s: %(message)s')
hdlr.setFormatter(fmt)
logging.root.addHandler(hdlr)
else:
logging.basicConfig(format='%(name)s: %(message)s', level=logging.INFO)
# Initialize the refactoring tool
rt = RefactoringTool(options)
# Refactor all files and directories passed as arguments
if not rt.errors:
rt.refactor_args(args)
rt.summarize()
# Return error status (0 if rt.errors is zero)
return int(bool(rt.errors))
def get_all_fix_names():
"""Return a sorted list of all available fix names."""
fix_names = []
names = os.listdir(os.path.dirname(fixes.__file__))
names.sort()
for name in names:
if name.startswith("fix_") and name.endswith(".py"):
fix_names.append(name[4:-3])
fix_names.sort()
return fix_names
def get_head_types(pat):
""" Accepts a pytree Pattern Node and returns a set
of the pattern types which will match first. """
if isinstance(pat, (pytree.NodePattern, pytree.LeafPattern)):
# NodePatters must either have no type and no content
# or a type and content -- so they don't get any farther
# Always return leafs
return set([pat.type])
if isinstance(pat, pytree.NegatedPattern):
if pat.content:
return get_head_types(pat.content)
return set([None]) # Negated Patterns don't have a type
if isinstance(pat, pytree.WildcardPattern):
# Recurse on each node in content
r = set()
for p in pat.content:
for x in p:
r.update(get_head_types(x))
return r
raise Exception("Oh no! I don't understand pattern %s" %(pat))
def get_headnode_dict(fixer_list):
""" Accepts a list of fixers and returns a dictionary
of head node type --> fixer list. """
head_nodes = defaultdict(list)
for fixer in fixer_list:
if not fixer.pattern:
head_nodes[None].append(fixer)
continue
for t in get_head_types(fixer.pattern):
head_nodes[t].append(fixer)
return head_nodes
class RefactoringTool(object):
def __init__(self, options):
"""Initializer.
The argument is an optparse.Values instance.
"""
self.options = options
self.errors = []
self.logger = logging.getLogger("RefactoringTool")
self.fixer_log = []
if self.options.print_function:
del pygram.python_grammar.keywords["print"]
self.driver = driver.Driver(pygram.python_grammar,
convert=pytree.convert,
logger=self.logger)
self.pre_order, self.post_order = self.get_fixers()
self.pre_order = get_headnode_dict(self.pre_order)
self.post_order = get_headnode_dict(self.post_order)
self.files = [] # List of files that were or should be modified
def get_fixers(self):
"""Inspects the options to load the requested patterns and handlers.
Returns:
(pre_order, post_order), where pre_order is the list of fixers that
want a pre-order AST traversal, and post_order is the list that want
post-order traversal.
"""
pre_order_fixers = []
post_order_fixers = []
fix_names = self.options.fix
if not fix_names or "all" in fix_names:
fix_names = get_all_fix_names()
for fix_name in fix_names:
try:
mod = __import__("lib2to3.fixes.fix_" + fix_name, {}, {}, ["*"])
except ImportError:
self.log_error("Can't find transformation %s", fix_name)
continue
parts = fix_name.split("_")
class_name = "Fix" + "".join([p.title() for p in parts])
try:
fix_class = getattr(mod, class_name)
except AttributeError:
self.log_error("Can't find fixes.fix_%s.%s",
fix_name, class_name)
continue
try:
fixer = fix_class(self.options, self.fixer_log)
except Exception as err:
self.log_error("Can't instantiate fixes.fix_%s.%s()",
fix_name, class_name, exc_info=True)
continue
if fixer.explicit and fix_name not in self.options.fix:
self.log_message("Skipping implicit fixer: %s", fix_name)
continue
if self.options.verbose:
self.log_message("Adding transformation: %s", fix_name)
if fixer.order == "pre":
pre_order_fixers.append(fixer)
elif fixer.order == "post":
post_order_fixers.append(fixer)
else:
raise ValueError("Illegal fixer order: %r" % fixer.order)
pre_order_fixers.sort(key=lambda x: x.run_order)
post_order_fixers.sort(key=lambda x: x.run_order)
return (pre_order_fixers, post_order_fixers)
def log_error(self, msg, *args, **kwds):
"""Increments error count and log a message."""
self.errors.append((msg, args, kwds))
self.logger.error(msg, *args, **kwds)
def log_message(self, msg, *args):
"""Hook to log a message."""
if args:
msg = msg % args
self.logger.info(msg)
def refactor_args(self, args):
"""Refactors files and directories from an argument list."""
for arg in args:
if arg == "-":
self.refactor_stdin()
elif os.path.isdir(arg):
self.refactor_dir(arg)
else:
self.refactor_file(arg)
def refactor_dir(self, arg):
"""Descends down a directory and refactor every Python file found.
Python files are assumed to have a .py extension.
Files and subdirectories starting with '.' are skipped.
"""
for dirpath, dirnames, filenames in os.walk(arg):
if self.options.verbose:
self.log_message("Descending into %s", dirpath)
dirnames.sort()
filenames.sort()
for name in filenames:
if not name.startswith(".") and name.endswith("py"):
fullname = os.path.join(dirpath, name)
self.refactor_file(fullname)
# Modify dirnames in-place to remove subdirs with leading dots
dirnames[:] = [dn for dn in dirnames if not dn.startswith(".")]
def refactor_file(self, filename):
"""Refactors a file."""
try:
f = open(filename)
except IOError as err:
self.log_error("Can't open %s: %s", filename, err)
return
try:
input = f.read() + "\n" # Silence certain parse errors
finally:
f.close()
if self.options.doctests_only:
if self.options.verbose:
self.log_message("Refactoring doctests in %s", filename)
output = self.refactor_docstring(input, filename)
if output != input:
self.write_file(output, filename, input)
elif self.options.verbose:
self.log_message("No doctest changes in %s", filename)
else:
tree = self.refactor_string(input, filename)
if tree and tree.was_changed:
# The [:-1] is to take off the \n we added earlier
self.write_file(str(tree)[:-1], filename)
elif self.options.verbose:
self.log_message("No changes in %s", filename)
def refactor_string(self, data, name):
"""Refactor a given input string.
Args:
data: a string holding the code to be refactored.
name: a human-readable name for use in error/log messages.
Returns:
An AST corresponding to the refactored input stream; None if
there were errors during the parse.
"""
try:
tree = self.driver.parse_string(data,1)
except Exception as err:
self.log_error("Can't parse %s: %s: %s",
name, err.__class__.__name__, err)
return
if self.options.verbose:
self.log_message("Refactoring %s", name)
self.refactor_tree(tree, name)
return tree
def refactor_stdin(self):
if self.options.write:
self.log_error("Can't write changes back to stdin")
return
input = sys.stdin.read()
if self.options.doctests_only:
if self.options.verbose:
self.log_message("Refactoring doctests in stdin")
output = self.refactor_docstring(input, "<stdin>")
if output != input:
self.write_file(output, "<stdin>", input)
elif self.options.verbose:
self.log_message("No doctest changes in stdin")
else:
tree = self.refactor_string(input, "<stdin>")
if tree and tree.was_changed:
self.write_file(str(tree), "<stdin>", input)
elif self.options.verbose:
self.log_message("No changes in stdin")
def refactor_tree(self, tree, name):
"""Refactors a parse tree (modifying the tree in place).
Args:
tree: a pytree.Node instance representing the root of the tree
to be refactored.
name: a human-readable name for this tree.
Returns:
True if the tree was modified, False otherwise.
"""
# Two calls to chain are required because pre_order.values()
# will be a list of lists of fixers:
# [[<fixer ...>, <fixer ...>], [<fixer ...>]]
all_fixers = chain(chain(*self.pre_order.values()),\
chain(*self.post_order.values()))
for fixer in all_fixers:
fixer.start_tree(tree, name)
self.traverse_by(self.pre_order, tree.pre_order())
self.traverse_by(self.post_order, tree.post_order())
for fixer in all_fixers:
fixer.finish_tree(tree, name)
return tree.was_changed
def traverse_by(self, fixers, traversal):
"""Traverse an AST, applying a set of fixers to each node.
This is a helper method for refactor_tree().
Args:
fixers: a list of fixer instances.
traversal: a generator that yields AST nodes.
Returns:
None
"""
if not fixers:
return
for node in traversal:
for fixer in fixers[node.type] + fixers[None]:
results = fixer.match(node)
if results:
new = fixer.transform(node, results)
if new is not None and (new != node or
str(new) != str(node)):
node.replace(new)
node = new
def write_file(self, new_text, filename, old_text=None):
"""Writes a string to a file.
If there are no changes, this is a no-op.
Otherwise, it first shows a unified diff between the old text
and the new text, and then rewrites the file; the latter is
only done if the write option is set.
"""
self.files.append(filename)
if old_text is None:
try:
f = open(filename, "r")
except IOError as err:
self.log_error("Can't read %s: %s", filename, err)
return
try:
old_text = f.read()
finally:
f.close()
if old_text == new_text:
if self.options.verbose:
self.log_message("No changes to %s", filename)
return
diff_texts(old_text, new_text, filename)
if not self.options.write:
if self.options.verbose:
self.log_message("Not writing changes to %s", filename)
return
backup = filename + ".bak"
if os.path.lexists(backup):
try:
os.remove(backup)
except os.error as err:
self.log_message("Can't remove backup %s", backup)
try:
os.rename(filename, backup)
except os.error as err:
self.log_message("Can't rename %s to %s", filename, backup)
try:
f = open(filename, "w")
except os.error as err:
self.log_error("Can't create %s: %s", filename, err)
return
try:
try:
f.write(new_text)
except os.error as err:
self.log_error("Can't write %s: %s", filename, err)
finally:
f.close()
if self.options.verbose:
self.log_message("Wrote changes to %s", filename)
PS1 = ">>> "
PS2 = "... "
def refactor_docstring(self, input, filename):
"""Refactors a docstring, looking for doctests.
This returns a modified version of the input string. It looks
for doctests, which start with a ">>>" prompt, and may be
continued with "..." prompts, as long as the "..." is indented
the same as the ">>>".
(Unfortunately we can't use the doctest module's parser,
since, like most parsers, it is not geared towards preserving
the original source.)
"""
result = []
block = None
block_lineno = None
indent = None
lineno = 0
for line in input.splitlines(True):
lineno += 1
if line.lstrip().startswith(self.PS1):
if block is not None:
result.extend(self.refactor_doctest(block, block_lineno,
indent, filename))
block_lineno = lineno
block = [line]
i = line.find(self.PS1)
indent = line[:i]
elif (indent is not None and
(line.startswith(indent + self.PS2) or
line == indent + self.PS2.rstrip() + "\n")):
block.append(line)
else:
if block is not None:
result.extend(self.refactor_doctest(block, block_lineno,
indent, filename))
block = None
indent = None
result.append(line)
if block is not None:
result.extend(self.refactor_doctest(block, block_lineno,
indent, filename))
return "".join(result)
def refactor_doctest(self, block, lineno, indent, filename):
"""Refactors one doctest.
A doctest is given as a block of lines, the first of which starts
with ">>>" (possibly indented), while the remaining lines start
with "..." (identically indented).
"""
try:
tree = self.parse_block(block, lineno, indent)
except Exception as err:
if self.options.verbose:
for line in block:
self.log_message("Source: %s", line.rstrip("\n"))
self.log_error("Can't parse docstring in %s line %s: %s: %s",
filename, lineno, err.__class__.__name__, err)
return block
if self.refactor_tree(tree, filename):
new = str(tree).splitlines(True)
# Undo the adjustment of the line numbers in wrap_toks() below.
clipped, new = new[:lineno-1], new[lineno-1:]
assert clipped == ["\n"] * (lineno-1), clipped
if not new[-1].endswith("\n"):
new[-1] += "\n"
block = [indent + self.PS1 + new.pop(0)]
if new:
block += [indent + self.PS2 + line for line in new]
return block
def summarize(self):
if self.options.write:
were = "were"
else:
were = "need to be"
if not self.files:
self.log_message("No files %s modified.", were)
else:
self.log_message("Files that %s modified:", were)
for file in self.files:
self.log_message(file)
if self.fixer_log:
self.log_message("Warnings/messages while refactoring:")
for message in self.fixer_log:
self.log_message(message)
if self.errors:
if len(self.errors) == 1:
self.log_message("There was 1 error:")
else:
self.log_message("There were %d errors:", len(self.errors))
for msg, args, kwds in self.errors:
self.log_message(msg, *args, **kwds)
def parse_block(self, block, lineno, indent):
"""Parses a block into a tree.
This is necessary to get correct line number / offset information
in the parser diagnostics and embedded into the parse tree.
"""
return self.driver.parse_tokens(self.wrap_toks(block, lineno, indent))
def wrap_toks(self, block, lineno, indent):
"""Wraps a tokenize stream to systematically modify start/end."""
tokens = tokenize.generate_tokens(self.gen_lines(block, indent).__next__)
for type, value, (line0, col0), (line1, col1), line_text in tokens:
line0 += lineno - 1
line1 += lineno - 1
# Don't bother updating the columns; this is too complicated
# since line_text would also have to be updated and it would
# still break for tokens spanning lines. Let the user guess
# that the column numbers for doctests are relative to the
# end of the prompt string (PS1 or PS2).
yield type, value, (line0, col0), (line1, col1), line_text
def gen_lines(self, block, indent):
"""Generates lines as expected by tokenize from a list of lines.
This strips the first len(indent + self.PS1) characters off each line.
"""
prefix1 = indent + self.PS1
prefix2 = indent + self.PS2
prefix = prefix1
for line in block:
if line.startswith(prefix):
yield line[len(prefix):]
elif line == prefix.rstrip() + "\n":
yield "\n"
else:
raise AssertionError("line=%r, prefix=%r" % (line, prefix))
prefix = prefix2
while True:
yield ""
def diff_texts(a, b, filename):
"""Prints a unified diff of two strings."""
a = a.splitlines()
b = b.splitlines()
for line in difflib.unified_diff(a, b, filename, filename,
"(original)", "(refactored)",
lineterm=""):
print(line)
if __name__ == "__main__":
sys.exit(main())