Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_threadedtempfile.py @ 87690:66c1c9f32567 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859
author Radomir Dopieralski <devel@sheep.art.pl>
date Tue, 03 Dec 2013 16:02:48 +0100
parents 3c9232a684ae
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
1 """
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
2 Create and delete FILES_PER_THREAD temp files (via tempfile.TemporaryFile)
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
3 in each of NUM_THREADS threads, recording the number of successes and
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
4 failures. A failure is a bug in tempfile, and may be due to:
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
5
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
6 + Trying to create more than one tempfile with the same name.
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
7 + Trying to delete a tempfile that doesn't still exist.
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
8 + Something we've never seen before.
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
9
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
10 By default, NUM_THREADS == 20 and FILES_PER_THREAD == 50. This is enough to
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
11 create about 150 failures per run under Win98SE in 2.0, and runs pretty
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
12 quickly. Guido reports needing to boost FILES_PER_THREAD to 500 before
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
13 provoking a 2.0 failure under Linux.
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
14 """
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
15
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
16 NUM_THREADS = 20
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
17 FILES_PER_THREAD = 50
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
18
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
19 import tempfile
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
20
61073
493bb4585e38 Merged revisions 80552-80556,80564-80566,80568-80571 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 49024
diff changeset
21 from test.support import threading_setup, threading_cleanup, run_unittest, import_module
493bb4585e38 Merged revisions 80552-80556,80564-80566,80568-80571 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 49024
diff changeset
22 threading = import_module('threading')
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
23 import unittest
42501
f6069f9dd52d SF patch# 1770008 by Christian Heimes (plus some extras).
Guido van Rossum <guido@python.org>
parents: 41554
diff changeset
24 import io
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
25 from traceback import print_exc
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
26
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
27 startEvent = threading.Event()
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
28
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
29 class TempFileGreedy(threading.Thread):
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
30 error_count = 0
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
31 ok_count = 0
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
32
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
33 def run(self):
42501
f6069f9dd52d SF patch# 1770008 by Christian Heimes (plus some extras).
Guido van Rossum <guido@python.org>
parents: 41554
diff changeset
34 self.errors = io.StringIO()
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
35 startEvent.wait()
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
36 for i in range(FILES_PER_THREAD):
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
37 try:
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
38 f = tempfile.TemporaryFile("w+b")
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
39 f.close()
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
40 except:
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
41 self.error_count += 1
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
42 print_exc(file=self.errors)
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
43 else:
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
44 self.ok_count += 1
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
45
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
46
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
47 class ThreadedTempFileTest(unittest.TestCase):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
48 def test_main(self):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
49 threads = []
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
50 thread_info = threading_setup()
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
51
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
52 for i in range(NUM_THREADS):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
53 t = TempFileGreedy()
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
54 threads.append(t)
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
55 t.start()
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
56
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
57 startEvent.set()
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
58
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
59 ok = 0
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
60 errors = []
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
61 for t in threads:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
62 t.join()
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
63 ok += t.ok_count
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
64 if t.error_count:
49024
a6620343e3db fix small typo
Benjamin Peterson <benjamin@python.org>
parents: 47669
diff changeset
65 errors.append(str(t.name) + str(t.errors.getvalue()))
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
66
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
67 threading_cleanup(*thread_info)
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
68
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
69 msg = "Errors: errors %d ok %d\n%s" % (len(errors), ok,
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
70 '\n'.join(errors))
66340
2beb25369289 Merged revisions 86596 via svnmerge from
Ezio Melotti <ezio.melotti@gmail.com>
parents: 49024
diff changeset
71 self.assertEqual(errors, [], msg)
2beb25369289 Merged revisions 86596 via svnmerge from
Ezio Melotti <ezio.melotti@gmail.com>
parents: 49024
diff changeset
72 self.assertEqual(ok, NUM_THREADS * FILES_PER_THREAD)
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
73
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
74 def test_main():
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
75 run_unittest(ThreadedTempFileTest)
25427
29f6160ec0bd SF bug 613233: test_threadedtempfile hangs
Tim Peters <tim.peters@gmail.com>
parents: 24364
diff changeset
76
15840
7d5025bba037 Guido found a brand new race in tempfile on Linux, due to Linux changing
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
77 if __name__ == "__main__":
25427
29f6160ec0bd SF bug 613233: test_threadedtempfile hangs
Tim Peters <tim.peters@gmail.com>
parents: 24364
diff changeset
78 test_main()