Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_threaded_import.py @ 87690:66c1c9f32567 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859
author Radomir Dopieralski <devel@sheep.art.pl>
date Tue, 03 Dec 2013 16:02:48 +0100
parents 1b8f08c4efd5
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
17897
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
1 # This is a variant of the very old (early 90's) file
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
2 # Demo/threads/bug.py. It simply provokes a number of threads into
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
3 # trying to import the same module "at the same time".
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
4 # There are no pleasant failure modes -- most likely is that Python
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
5 # complains several times about module random having no attribute
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
6 # randrange, and then Python hangs.
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
7
84167
1b8f08c4efd5 Issue #17177: The imp module is pending deprecation.
Brett Cannon <brett@python.org>
parents: 80939
diff changeset
8 import _imp as imp
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
9 import os
80939
4332b51076e2 Call importlib.invalidate_caches() after creating module files in test_threaded_import.
Antoine Pitrou <solipsis@pitrou.net>
parents: 80937
diff changeset
10 import importlib
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
11 import sys
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
12 import time
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
13 import shutil
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
14 import unittest
71370
9242333a8d0d test_threaded_import must clean up after itself
Antoine Pitrou <solipsis@pitrou.net>
parents: 70530
diff changeset
15 from test.support import (
77016
423a26e8ffbb Make the test completely clean up after itself.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77013
diff changeset
16 verbose, import_module, run_unittest, TESTFN, reap_threads, forget, unlink)
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
17 threading = import_module('threading')
17897
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
18
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
19 def task(N, done, done_tasks, errors):
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
20 try:
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
21 # We don't use modulefinder but still import it in order to stress
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
22 # importing of different modules from several threads.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
23 if len(done_tasks) % 2:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
24 import modulefinder
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
25 import random
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
26 else:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
27 import random
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
28 import modulefinder
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
29 # This will fail if random is not completely initialized
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
30 x = random.randrange(1, 3)
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
31 except Exception as e:
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
32 errors.append(e.with_traceback(None))
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
33 finally:
70530
5548197d1f22 Close #12028: Make threading._get_ident() public, rename it to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68257
diff changeset
34 done_tasks.append(threading.get_ident())
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
35 finished = len(done_tasks) == N
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
36 if finished:
65280
1e72abf073c5 Merged revisions 85444 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 64238
diff changeset
37 done.set()
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
38
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
39 # Create a circular import structure: A -> C -> B -> D -> A
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
40 # NOTE: `time` is already loaded and therefore doesn't threaten to deadlock.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
41
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
42 circular_imports_modules = {
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
43 'A': """if 1:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
44 import time
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
45 time.sleep(%(delay)s)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
46 x = 'a'
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
47 import C
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
48 """,
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
49 'B': """if 1:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
50 import time
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
51 time.sleep(%(delay)s)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
52 x = 'b'
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
53 import D
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
54 """,
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
55 'C': """import B""",
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
56 'D': """import A""",
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
57 }
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
58
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
59 class Finder:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
60 """A dummy finder to detect concurrent access to its find_module()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
61 method."""
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
62
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
63 def __init__(self):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
64 self.numcalls = 0
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
65 self.x = 0
71370
9242333a8d0d test_threaded_import must clean up after itself
Antoine Pitrou <solipsis@pitrou.net>
parents: 70530
diff changeset
66 self.lock = threading.Lock()
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
67
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
68 def find_module(self, name, path=None):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
69 # Simulate some thread-unsafe behaviour. If calls to find_module()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
70 # are properly serialized, `x` will end up the same as `numcalls`.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
71 # Otherwise not.
80935
1ee9cf5a5351 Add sanity assertions in some import lock code (issue #15599).
Antoine Pitrou <solipsis@pitrou.net>
parents: 79383
diff changeset
72 assert imp.lock_held()
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
73 with self.lock:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
74 self.numcalls += 1
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
75 x = self.x
80937
67d6bb557d67 Try to make test_threaded_import faster and safer (issue #15599).
Antoine Pitrou <solipsis@pitrou.net>
parents: 80935
diff changeset
76 time.sleep(0.01)
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
77 self.x = x + 1
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
78
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
79 class FlushingFinder:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
80 """A dummy finder which flushes sys.path_importer_cache when it gets
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
81 called."""
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
82
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
83 def find_module(self, name, path=None):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
84 sys.path_importer_cache.clear()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
85
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
86
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
87 class ThreadedImportTests(unittest.TestCase):
17897
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
88
63045
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
89 def setUp(self):
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
90 self.old_random = sys.modules.pop('random', None)
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
91
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
92 def tearDown(self):
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
93 # If the `random` module was already initialized, we restore the
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
94 # old module at the end so that pickling tests don't fail.
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
95 # See http://bugs.python.org/issue3657#msg110461
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
96 if self.old_random is not None:
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
97 sys.modules['random'] = self.old_random
7cbf8c4e2727 Merged revisions 82919 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63024
diff changeset
98
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
99 def check_parallel_module_init(self):
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
100 if imp.lock_held():
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
101 # This triggers on, e.g., from test import autotest.
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
102 raise unittest.SkipTest("can't run when import lock is held")
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
103
65280
1e72abf073c5 Merged revisions 85444 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 64238
diff changeset
104 done = threading.Event()
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
105 for N in (20, 50) * 3:
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
106 if verbose:
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
107 print("Trying", N, "threads ...", end=' ')
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
108 # Make sure that random and modulefinder get reimported freshly
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
109 for modname in ['random', 'modulefinder']:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
110 try:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
111 del sys.modules[modname]
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
112 except KeyError:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
113 pass
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
114 errors = []
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
115 done_tasks = []
65280
1e72abf073c5 Merged revisions 85444 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 64238
diff changeset
116 done.clear()
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
117 for i in range(N):
71370
9242333a8d0d test_threaded_import must clean up after itself
Antoine Pitrou <solipsis@pitrou.net>
parents: 70530
diff changeset
118 t = threading.Thread(target=task,
9242333a8d0d test_threaded_import must clean up after itself
Antoine Pitrou <solipsis@pitrou.net>
parents: 70530
diff changeset
119 args=(N, done, done_tasks, errors,))
9242333a8d0d test_threaded_import must clean up after itself
Antoine Pitrou <solipsis@pitrou.net>
parents: 70530
diff changeset
120 t.start()
80937
67d6bb557d67 Try to make test_threaded_import faster and safer (issue #15599).
Antoine Pitrou <solipsis@pitrou.net>
parents: 80935
diff changeset
121 self.assertTrue(done.wait(60))
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
122 self.assertFalse(errors)
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
123 if verbose:
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
124 print("OK.")
17897
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
125
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
126 def test_parallel_module_init(self):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
127 self.check_parallel_module_init()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
128
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
129 def test_parallel_meta_path(self):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
130 finder = Finder()
76578
c18256de00bb Issue #14605: Insert to the front of sys.meta_path, don't append.
Brett Cannon <brett@python.org>
parents: 76562
diff changeset
131 sys.meta_path.insert(0, finder)
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
132 try:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
133 self.check_parallel_module_init()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
134 self.assertGreater(finder.numcalls, 0)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
135 self.assertEqual(finder.x, finder.numcalls)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
136 finally:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
137 sys.meta_path.remove(finder)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
138
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
139 def test_parallel_path_hooks(self):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
140 # Here the Finder instance is only used to check concurrent calls
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
141 # to path_hook().
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
142 finder = Finder()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
143 # In order for our path hook to be called at each import, we need
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
144 # to flush the path_importer_cache, which we do by registering a
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
145 # dedicated meta_path entry.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
146 flushing_finder = FlushingFinder()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
147 def path_hook(path):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
148 finder.find_module('')
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
149 raise ImportError
76562
8dab93ec19de Issue #14605: Insert to the front of sys.path_hooks instead of appending.
Brett Cannon <brett@python.org>
parents: 71370
diff changeset
150 sys.path_hooks.insert(0, path_hook)
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
151 sys.meta_path.append(flushing_finder)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
152 try:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
153 # Flush the cache a first time
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
154 flushing_finder.find_module('')
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
155 numtests = self.check_parallel_module_init()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
156 self.assertGreater(finder.numcalls, 0)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
157 self.assertEqual(finder.x, finder.numcalls)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
158 finally:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
159 sys.meta_path.remove(flushing_finder)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
160 sys.path_hooks.remove(path_hook)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
161
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
162 def test_import_hangers(self):
37908
c7ef92b9045c SF bug #1473760 TempFile can hang on Windows.
Tim Peters <tim.peters@gmail.com>
parents: 32918
diff changeset
163 # In case this test is run again, make sure the helper module
c7ef92b9045c SF bug #1473760 TempFile can hang on Windows.
Tim Peters <tim.peters@gmail.com>
parents: 32918
diff changeset
164 # gets loaded from scratch again.
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
165 try:
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
166 del sys.modules['test.threaded_import_hangers']
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
167 except KeyError:
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
168 pass
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
169 import test.threaded_import_hangers
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
170 self.assertFalse(test.threaded_import_hangers.errors)
19596
1be18ef869a4 Add a new function imp.lock_held(), and use it to skip test_threaded_import
Tim Peters <tim.peters@gmail.com>
parents: 19564
diff changeset
171
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
172 def test_circular_imports(self):
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
173 # The goal of this test is to exercise implementations of the import
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
174 # lock which use a per-module lock, rather than a global lock.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
175 # In these implementations, there is a possible deadlock with
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
176 # circular imports, for example:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
177 # - thread 1 imports A (grabbing the lock for A) which imports B
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
178 # - thread 2 imports B (grabbing the lock for B) which imports A
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
179 # Such implementations should be able to detect such situations and
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
180 # resolve them one way or the other, without freezing.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
181 # NOTE: our test constructs a slightly less trivial import cycle,
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
182 # in order to better stress the deadlock avoidance mechanism.
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
183 delay = 0.5
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
184 os.mkdir(TESTFN)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
185 self.addCleanup(shutil.rmtree, TESTFN)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
186 sys.path.insert(0, TESTFN)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
187 self.addCleanup(sys.path.remove, TESTFN)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
188 for name, contents in circular_imports_modules.items():
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
189 contents = contents % {'delay': delay}
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
190 with open(os.path.join(TESTFN, name + ".py"), "wb") as f:
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
191 f.write(contents.encode('utf-8'))
77013
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
192 self.addCleanup(forget, name)
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
193
80939
4332b51076e2 Call importlib.invalidate_caches() after creating module files in test_threaded_import.
Antoine Pitrou <solipsis@pitrou.net>
parents: 80937
diff changeset
194 importlib.invalidate_caches()
64238
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
195 results = []
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
196 def import_ab():
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
197 import A
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
198 results.append(getattr(A, 'x', None))
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
199 def import_ba():
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
200 import B
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
201 results.append(getattr(B, 'x', None))
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
202 t1 = threading.Thread(target=import_ab)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
203 t2 = threading.Thread(target=import_ba)
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
204 t1.start()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
205 t2.start()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
206 t1.join()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
207 t2.join()
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
208 self.assertEqual(set(results), {'a', 'b'})
716d3d102c0a Merged revisions 84258,84268 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 63045
diff changeset
209
77013
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
210 def test_side_effect_import(self):
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
211 code = """if 1:
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
212 import threading
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
213 def target():
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
214 import random
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
215 t = threading.Thread(target=target)
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
216 t.start()
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
217 t.join()"""
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
218 sys.path.insert(0, os.curdir)
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
219 self.addCleanup(sys.path.remove, os.curdir)
77016
423a26e8ffbb Make the test completely clean up after itself.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77013
diff changeset
220 filename = TESTFN + ".py"
423a26e8ffbb Make the test completely clean up after itself.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77013
diff changeset
221 with open(filename, "wb") as f:
77013
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
222 f.write(code.encode('utf-8'))
77016
423a26e8ffbb Make the test completely clean up after itself.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77013
diff changeset
223 self.addCleanup(unlink, filename)
77013
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
224 self.addCleanup(forget, TESTFN)
80939
4332b51076e2 Call importlib.invalidate_caches() after creating module files in test_threaded_import.
Antoine Pitrou <solipsis@pitrou.net>
parents: 80937
diff changeset
225 importlib.invalidate_caches()
77013
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
226 __import__(TESTFN)
edb9ce3a6c2e Issue #9260: A finer-grained import lock.
Antoine Pitrou <solipsis@pitrou.net>
parents: 76578
diff changeset
227
19596
1be18ef869a4 Add a new function imp.lock_held(), and use it to skip test_threaded_import
Tim Peters <tim.peters@gmail.com>
parents: 19564
diff changeset
228
71370
9242333a8d0d test_threaded_import must clean up after itself
Antoine Pitrou <solipsis@pitrou.net>
parents: 70530
diff changeset
229 @reap_threads
63024
3805ffe8ff38 Merged revisions 82885 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 52389
diff changeset
230 def test_main():
78779
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
231 old_switchinterval = None
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
232 try:
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
233 old_switchinterval = sys.getswitchinterval()
79383
484a4b9349af Issue #15599: Increase the switch interval. Several systems cannot handle
Stefan Krah <skrah@bytereef.org>
parents: 79018
diff changeset
234 sys.setswitchinterval(1e-5)
78779
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
235 except AttributeError:
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
236 pass
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
237 try:
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
238 run_unittest(ThreadedImportTests)
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
239 finally:
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
240 if old_switchinterval is not None:
7fa6336e9864 Issue #15781: Fix two small race conditions in import's module locking.
Antoine Pitrou <solipsis@pitrou.net>
parents: 77016
diff changeset
241 sys.setswitchinterval(old_switchinterval)
17897
cc984ed26586 New test adapted from the ancient Demo/threads/bug.py.
Tim Peters <tim.peters@gmail.com>
parents:
diff changeset
242
17913
46909f22170c Implementing an idea from Guido on the checkins list:
Tim Peters <tim.peters@gmail.com>
parents: 17897
diff changeset
243 if __name__ == "__main__":
46909f22170c Implementing an idea from Guido on the checkins list:
Tim Peters <tim.peters@gmail.com>
parents: 17897
diff changeset
244 test_main()