cpython-lru_cache-weakref: Tools/scripts/pysource.py annotate

annotate Tools/scripts/pysource.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	8ace059cdffd
children

rev	line source
59522 fcfe324663a6 convert shebang lines: python -> python3 Benjamin Peterson <benjamin@python.org> parents: 45471 diff changeset	1 #!/usr/bin/env python3
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	2
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	3 """\
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	4 List python source files.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	5
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	6 There are three functions to check whether a file is a Python source, listed
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	7 here with increasing complexity:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	8
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	9 - has_python_ext() checks whether a file name ends in '.py[w]'.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	10 - look_like_python() checks whether the file is not binary and either has
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	11 the '.py[w]' extension or the first line contains the word 'python'.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	12 - can_be_compiled() checks whether the file can be compiled by compile().
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	13
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	14 The file also must be of appropriate size - not bigger than a megabyte.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	15
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	16 walk_python_files() recursively lists all Python files under the given directories.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	17 """
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 35465 diff changeset	18 __author__ = "Oleg Broytmann, Georg Brandl"
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	19
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	20 __all__ = ["has_python_ext", "looks_like_python", "can_be_compiled", "walk_python_files"]
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	21
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	22
45471 7dd2c1746267 Merged revisions 60990-61002 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 42427 diff changeset	23 import os, re
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	24
78375 67d36e8ddcfc Fix findnocoding.p and pysource.py scripts Victor Stinner <victor.stinner@gmail.com> parents: 70404 diff changeset	25 binary_re = re.compile(br'[\x00-\x08\x0E-\x1F\x7F]')
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	26
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	27 debug = False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	28
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	29 def print_debug(msg):
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	30 if debug: print(msg)
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	31
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	32
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	33 def _open(fullpath):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	34 try:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	35 size = os.stat(fullpath).st_size
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	36 except OSError as err: # Permission denied - ignore the file
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	37 print_debug("%s: permission denied: %s" % (fullpath, err))
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	38 return None
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	39
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	40 if size > 1024*1024: # too big
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	41 print_debug("%s: the file is too big: %d bytes" % (fullpath, size))
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	42 return None
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	43
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	44 try:
78375 67d36e8ddcfc Fix findnocoding.p and pysource.py scripts Victor Stinner <victor.stinner@gmail.com> parents: 70404 diff changeset	45 return open(fullpath, "rb")
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	46 except IOError as err: # Access denied, or a special file - ignore it
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	47 print_debug("%s: access denied: %s" % (fullpath, err))
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	48 return None
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	49
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	50 def has_python_ext(fullpath):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	51 return fullpath.endswith(".py") or fullpath.endswith(".pyw")
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	52
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	53 def looks_like_python(fullpath):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	54 infile = _open(fullpath)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	55 if infile is None:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	56 return False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	57
78376 8ace059cdffd Cleanup findnocoding.py and pysource.py scripts (with infile/infile.close) Victor Stinner <victor.stinner@gmail.com> parents: 78375 diff changeset	58 with infile:
8ace059cdffd Cleanup findnocoding.py and pysource.py scripts (with infile/infile.close) Victor Stinner <victor.stinner@gmail.com> parents: 78375 diff changeset	59 line = infile.readline()
35465 3e5c07faab11 Whitespace normalization (via reindent.py). Tim Peters <tim.peters@gmail.com> parents: 35424 diff changeset	60
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	61 if binary_re.search(line):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	62 # file appears to be binary
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	63 print_debug("%s: appears to be binary" % fullpath)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	64 return False
35465 3e5c07faab11 Whitespace normalization (via reindent.py). Tim Peters <tim.peters@gmail.com> parents: 35424 diff changeset	65
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	66 if fullpath.endswith(".py") or fullpath.endswith(".pyw"):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	67 return True
78375 67d36e8ddcfc Fix findnocoding.p and pysource.py scripts Victor Stinner <victor.stinner@gmail.com> parents: 70404 diff changeset	68 elif b"python" in line:
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	69 # disguised Python script (e.g. CGI)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	70 return True
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	71
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	72 return False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	73
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	74 def can_be_compiled(fullpath):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	75 infile = _open(fullpath)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	76 if infile is None:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	77 return False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	78
78376 8ace059cdffd Cleanup findnocoding.py and pysource.py scripts (with infile/infile.close) Victor Stinner <victor.stinner@gmail.com> parents: 78375 diff changeset	79 with infile:
8ace059cdffd Cleanup findnocoding.py and pysource.py scripts (with infile/infile.close) Victor Stinner <victor.stinner@gmail.com> parents: 78375 diff changeset	80 code = infile.read()
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	81
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	82 try:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	83 compile(code, fullpath, "exec")
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	84 except Exception as err:
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	85 print_debug("%s: cannot compile: %s" % (fullpath, err))
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	86 return False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	87
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	88 return True
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	89
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	90
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	91 def walk_python_files(paths, is_python=looks_like_python, exclude_dirs=None):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	92 """\
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	93 Recursively yield all Python source files below the given paths.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	94
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	95 paths: a list of files and/or directories to be checked.
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	96 is_python: a function that takes a file name and checks whether it is a
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	97 Python source file
35465 3e5c07faab11 Whitespace normalization (via reindent.py). Tim Peters <tim.peters@gmail.com> parents: 35424 diff changeset	98 exclude_dirs: a list of directory base names that should be excluded in
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	99 the search
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	100 """
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	101 if exclude_dirs is None:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	102 exclude_dirs=[]
35465 3e5c07faab11 Whitespace normalization (via reindent.py). Tim Peters <tim.peters@gmail.com> parents: 35424 diff changeset	103
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	104 for path in paths:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	105 print_debug("testing: %s" % path)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	106 if os.path.isfile(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	107 if is_python(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	108 yield path
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	109 elif os.path.isdir(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	110 print_debug(" it is a directory")
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	111 for dirpath, dirnames, filenames in os.walk(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	112 for exclude in exclude_dirs:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	113 if exclude in dirnames:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	114 dirnames.remove(exclude)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	115 for filename in filenames:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	116 fullpath = os.path.join(dirpath, filename)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	117 print_debug("testing: %s" % fullpath)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	118 if is_python(fullpath):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	119 yield fullpath
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	120 else:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	121 print_debug(" unknown type")
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	122
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	123
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	124 if __name__ == "__main__":
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	125 # Two simple examples/tests
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	126 for fullpath in walk_python_files(['.']):
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	127 print(fullpath)
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	128 print("----------")
35424 7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding Georg Brandl <georg@python.org> parents: diff changeset	129 for fullpath in walk_python_files(['.'], is_python=can_be_compiled):
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	130 print(fullpath)

Mercurial > cpython-lru_cache-weakref

annotate Tools/scripts/pysource.py @ 87690:66c1c9f32567 default tip