annotate Tools/scripts/findnocoding.py @ 73323:dad02a080bbc

Improve byte-compilation in packaging to be independent of -O or -B. The code I fixed to comply with PEP 3147 still had one bug: When run under python -O, some paths for pyc files would be pyo, because I called imp.cache_from_source without explicit debug_override argument in some places, and under -O that would return .pyo (this is well explained in the imp docs). Now all code (util.byte_compile, build_py, install_lib) can create .pyo files according to options given by users, without interference from the calling Python’s own optimize mode. On a related topic, I also removed the code that prevented byte compilation under python -B. The rationale is that packaging gives control over the creation of pyc files to the user with its own explicit option, and the behavior should not be changed if the calling Python happens to run with -B for whatever reason. I will argue that this is a bug fix and ask to be allowed to backport this change to distutils. Finally, I moved one nugget of information about the --compile and --optimize options from the source into the doc. It clears up a misunderstanding that I (and maybe other people) had.
author Éric Araujo <merwok@netwok.org>
date Thu, 03 Nov 2011 05:08:28 +0100
parents 6d9cb906c85e
children 696cb524322a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
59522
fcfe324663a6 convert shebang lines: python -> python3
Benjamin Peterson <benjamin@python.org>
parents: 49633
diff changeset
1 #!/usr/bin/env python3
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
2
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
3 """List all those Python files that require a coding directive
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
4
70404
6d9cb906c85e Remove obsolete 'rU' mode in Tools/scripts too (see be5b8d1ded34)
Éric Araujo <merwok@netwok.org>
parents: 59522
diff changeset
5 Usage: findnocoding.py dir1 [dir2...]
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
6 """
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
7
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 35465
diff changeset
8 __author__ = "Oleg Broytmann, Georg Brandl"
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
9
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
10 import sys, os, re, getopt
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
11
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
12 # our pysource module finds Python source files
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
13 try:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
14 import pysource
49633
bd0e9a9cd283 Merged revisions 67028,67040,67044,67046,67052,67065,67070,67077,67082 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 49591
diff changeset
15 except ImportError:
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
16 # emulate the module with a simple os.walk
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
17 class pysource:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
18 has_python_ext = looks_like_python = can_be_compiled = None
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
19 def walk_python_files(self, paths, *args, **kwargs):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
20 for path in paths:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
21 if os.path.isfile(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
22 yield path.endswith(".py")
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
23 elif os.path.isdir(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
24 for root, dirs, files in os.walk(path):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
25 for filename in files:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
26 if filename.endswith(".py"):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
27 yield os.path.join(root, filename)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
28 pysource = pysource()
35465
3e5c07faab11 Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents: 35424
diff changeset
29
3e5c07faab11 Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents: 35424
diff changeset
30
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
31 print("The pysource module is not available; "
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
32 "no sophisticated Python source file search will be done.", file=sys.stderr)
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
33
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
34
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
35 decl_re = re.compile(r"coding[=:]\s*([-\w.]+)")
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
36
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
37 def get_declaration(line):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
38 match = decl_re.search(line)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
39 if match:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
40 return match.group(1)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
41 return ''
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
42
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
43 def has_correct_encoding(text, codec):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
44 try:
47100
893a4dc17a91 Ran 2to3 over scripts directory.
Georg Brandl <georg@python.org>
parents: 42427
diff changeset
45 str(text, codec)
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
46 except UnicodeDecodeError:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
47 return False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
48 else:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
49 return True
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
50
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
51 def needs_declaration(fullpath):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
52 try:
70404
6d9cb906c85e Remove obsolete 'rU' mode in Tools/scripts too (see be5b8d1ded34)
Éric Araujo <merwok@netwok.org>
parents: 59522
diff changeset
53 infile = open(fullpath)
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
54 except IOError: # Oops, the file was removed - ignore it
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
55 return None
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
56
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
57 line1 = infile.readline()
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
58 line2 = infile.readline()
35465
3e5c07faab11 Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents: 35424
diff changeset
59
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
60 if get_declaration(line1) or get_declaration(line2):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
61 # the file does have an encoding declaration, so trust it
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
62 infile.close()
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
63 return False
35465
3e5c07faab11 Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents: 35424
diff changeset
64
49591
860737c96e97 default source encoding is now utf-8
Benjamin Peterson <benjamin@python.org>
parents: 47100
diff changeset
65 # check the whole file for non utf-8 characters
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
66 rest = infile.read()
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
67 infile.close()
35465
3e5c07faab11 Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents: 35424
diff changeset
68
49591
860737c96e97 default source encoding is now utf-8
Benjamin Peterson <benjamin@python.org>
parents: 47100
diff changeset
69 if has_correct_encoding(line1+line2+rest, "utf-8"):
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
70 return False
35465
3e5c07faab11 Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents: 35424
diff changeset
71
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
72 return True
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
73
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
74
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
75 usage = """Usage: %s [-cd] paths...
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
76 -c: recognize Python source files trying to compile them
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
77 -d: debug output""" % sys.argv[0]
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
78
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
79 try:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
80 opts, args = getopt.getopt(sys.argv[1:], 'cd')
40587
6b18a095c2e7 SF patch 1631942 by Collin Winter:
Guido van Rossum <guido@python.org>
parents: 40450
diff changeset
81 except getopt.error as msg:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
82 print(msg, file=sys.stderr)
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
83 print(usage, file=sys.stderr)
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
84 sys.exit(1)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
85
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
86 is_python = pysource.looks_like_python
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
87 debug = False
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
88
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
89 for o, a in opts:
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
90 if o == '-c':
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
91 is_python = pysource.can_be_compiled
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
92 elif o == '-d':
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
93 debug = True
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
94
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
95 if not args:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
96 print(usage, file=sys.stderr)
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
97 sys.exit(1)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
98
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
99 for fullpath in pysource.walk_python_files(args, is_python):
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
100 if debug:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
101 print("Testing for coding: %s" % fullpath)
35424
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
102 result = needs_declaration(fullpath)
7caabc749975 Patch [ 784089 ] A program to scan python files and list those require coding
Georg Brandl <georg@python.org>
parents:
diff changeset
103 if result:
42427
46625e6122e9 Convert print statements to function calls in Tools/.
Collin Winter <collinw@gmail.com>
parents: 40587
diff changeset
104 print(fullpath)