Mercurial > cpython-lru_cache-weakref

annotate Tools/scripts/find_recursionlimit.py @ 87690:66c1c9f32567 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859
author Radomir Dopieralski <devel@sheep.art.pl>
date Tue, 03 Dec 2013 16:02:48 +0100
parents 15a391919deb
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
59522
fcfe324663a6 convert shebang lines: python -> python3
Benjamin Peterson <benjamin@python.org>
parents: 56889
diff changeset
1 #! /usr/bin/env python3
56889
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
2 """Find the maximum recursion limit that prevents interpreter termination.
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
3
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
4 This script finds the maximum safe recursion limit on a particular
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
5 platform. If you need to change the recursion limit on your system,
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
6 this script will tell you a safe upper bound. To use the new limit,
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
7 call sys.setrecursionlimit().
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
8
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
9 This module implements several ways to create infinite recursion in
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
10 Python. Different implementations end up pushing different numbers of
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
11 C stack frames, depending on how many calls through Python's abstract
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
12 C API occur.
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
13
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
14 After each round of tests, it prints a message:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
15 "Limit of NNNN is fine".
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
16
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
17 The highest printed value of "NNNN" is therefore the highest potentially
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
18 safe limit for your system (which depends on the OS, architecture, but also
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
19 the compilation flags). Please note that it is practically impossible to
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
20 test all possible recursion paths in the interpreter, so the results of
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
21 this test should not be trusted blindly -- although they give a good hint
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
22 of which values are reasonable.
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
23
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
24 NOTE: When the C stack space allocated by your system is exceeded due
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
25 to excessive recursion, exact behaviour depends on the platform, although
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
26 the interpreter will always fail in a likely brutal way: either a
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
27 segmentation fault, a MemoryError, or just a silent abort.
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
28
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
29 NB: A program that does not use __methods__ can set a higher limit.
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
30 """
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
31
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
32 import sys
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
33 import itertools
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
34
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
35 class RecursiveBlowup1:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
36 def __init__(self):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
37 self.__init__()
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
38
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
39 def test_init():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
40 return RecursiveBlowup1()
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
41
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
42 class RecursiveBlowup2:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
43 def __repr__(self):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
44 return repr(self)
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
45
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
46 def test_repr():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
47 return repr(RecursiveBlowup2())
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
48
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
49 class RecursiveBlowup4:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
50 def __add__(self, x):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
51 return x + self
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
52
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
53 def test_add():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
54 return RecursiveBlowup4() + RecursiveBlowup4()
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
55
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
56 class RecursiveBlowup5:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
57 def __getattr__(self, attr):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
58 return getattr(self, attr)
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
59
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
60 def test_getattr():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
61 return RecursiveBlowup5().attr
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
62
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
63 class RecursiveBlowup6:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
64 def __getitem__(self, item):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
65 return self[item - 2] + self[item - 1]
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
66
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
67 def test_getitem():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
68 return RecursiveBlowup6()[5]
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
69
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
70 def test_recurse():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
71 return test_recurse()
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
72
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
73 def test_cpickle(_cache={}):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
74 import io
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
75 try:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
76 import _pickle
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
77 except ImportError:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
78 print("cannot import _pickle, skipped!")
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
79 return
67725
bcf030f73eee Merged revisions 88147 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56889
diff changeset
80 k, l = None, None
56889
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
81 for n in itertools.count():
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
82 try:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
83 l = _cache[n]
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
84 continue # Already tried and it works, let's save some time
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
85 except KeyError:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
86 for i in range(100):
67725
bcf030f73eee Merged revisions 88147 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56889
diff changeset
87 l = [k, l]
bcf030f73eee Merged revisions 88147 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56889
diff changeset
88 k = {i: l}
56889
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
89 _pickle.Pickler(io.BytesIO(), protocol=-1).dump(l)
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
90 _cache[n] = l
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
91
80241
cf2515d0328b Issue #5765: Also check the compiler when finding the recursion limit
Nick Coghlan <ncoghlan@gmail.com>
parents: 76106
diff changeset
92 def test_compiler_recursion():
cf2515d0328b Issue #5765: Also check the compiler when finding the recursion limit
Nick Coghlan <ncoghlan@gmail.com>
parents: 76106
diff changeset
93 # The compiler uses a scaling factor to support additional levels
cf2515d0328b Issue #5765: Also check the compiler when finding the recursion limit
Nick Coghlan <ncoghlan@gmail.com>
parents: 76106
diff changeset
94 # of recursion. This is a sanity check of that scaling to ensure
80929
15a391919deb Issue #16714: use 'raise' exceptions, don't 'throw'.
Andrew Svetlov <andrew.svetlov@gmail.com>
parents: 80241
diff changeset
95 # it still raises RuntimeError even at higher recursion limits
80241
cf2515d0328b Issue #5765: Also check the compiler when finding the recursion limit
Nick Coghlan <ncoghlan@gmail.com>
parents: 76106
diff changeset
96 compile("()" * (10 * sys.getrecursionlimit()), "<single>", "single")
cf2515d0328b Issue #5765: Also check the compiler when finding the recursion limit
Nick Coghlan <ncoghlan@gmail.com>
parents: 76106
diff changeset
97
56889
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
98 def check_limit(n, test_func_name):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
99 sys.setrecursionlimit(n)
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
100 if test_func_name.startswith("test_"):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
101 print(test_func_name[5:])
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
102 else:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
103 print(test_func_name)
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
104 test_func = globals()[test_func_name]
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
105 try:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
106 test_func()
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
107 # AttributeError can be raised because of the way e.g. PyDict_GetItem()
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
108 # silences all exceptions and returns NULL, which is usually interpreted
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
109 # as "missing attribute".
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
110 except (RuntimeError, AttributeError):
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
111 pass
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
112 else:
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
113 print("Yikes!")
029d69659b5f Merged revisions 75373 via svnmerge from
Georg Brandl <georg@python.org>
parents:
diff changeset
114
76106
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
115 if __name__ == '__main__':
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
116
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
117 limit = 1000
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
118 while 1:
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
119 check_limit(limit, "test_recurse")
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
120 check_limit(limit, "test_add")
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
121 check_limit(limit, "test_repr")
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
122 check_limit(limit, "test_init")
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
123 check_limit(limit, "test_getattr")
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
124 check_limit(limit, "test_getitem")
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
125 check_limit(limit, "test_cpickle")
80241
cf2515d0328b Issue #5765: Also check the compiler when finding the recursion limit
Nick Coghlan <ncoghlan@gmail.com>
parents: 76106
diff changeset
126 check_limit(limit, "test_compiler_recursion")
76106
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
127 print("Limit of %d is fine" % limit)
62dde5dd475e #14490, #14491: add 'sundry'-style import tests for Tools/scripts.
R David Murray <rdmurray@bitdance.com>
parents: 68257
diff changeset
128 limit = limit + 100