Mercurial > cpython-lru_cache-weakref

annotate Modules/gc_weakref.txt @ 54294:dba767bc3155 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
remove mention of old ctypes version
author Benjamin Peterson <benjamin@python.org>
date Sat, 23 May 2009 20:59:09 +0000
parents facb5a2f4b53
children 021e5bb297d1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
1 Intro
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
2 =====
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
3
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
4 The basic rule for dealing with weakref callbacks (and __del__ methods too,
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
5 for that matter) during cyclic gc:
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
6
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
7 Once gc has computed the set of unreachable objects, no Python-level
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
8 code can be allowed to access an unreachable object.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
9
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
10 If that can happen, then the Python code can resurrect unreachable objects
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
11 too, and gc can't detect that without starting over. Since gc eventually
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
12 runs tp_clear on all unreachable objects, if an unreachable object is
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
13 resurrected then tp_clear will eventually be called on it (or may already
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
14 have been called before resurrection). At best (and this has been an
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
15 historically common bug), tp_clear empties an instance's __dict__, and
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
16 "impossible" AttributeErrors result. At worst, tp_clear leaves behind an
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
17 insane object at the C level, and segfaults result (historically, most
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
18 often by setting a new-style class's mro pointer to NULL, after which
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
19 attribute lookups performed by the class can segfault).
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
20
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
21 OTOH, it's OK to run Python-level code that can't access unreachable
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
22 objects, and sometimes that's necessary. The chief example is the callback
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
23 attached to a reachable weakref W to an unreachable object O. Since O is
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
24 going away, and W is still alive, the callback must be invoked. Because W
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
25 is still alive, everything reachable from its callback is also reachable,
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
26 so it's also safe to invoke the callback (although that's trickier than it
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
27 sounds, since other reachable weakrefs to other unreachable objects may
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
28 still exist, and be accessible to the callback -- there are lots of painful
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
29 details like this covered in the rest of this file).
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
30
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
31 Python 2.4/2.3.5
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
32 ================
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
33
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
34 The "Before 2.3.3" section below turned out to be wrong in some ways, but
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
35 I'm leaving it as-is because it's more right than wrong, and serves as a
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
36 wonderful example of how painful analysis can miss not only the forest for
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
37 the trees, but also miss the trees for the aphids sucking the trees
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
38 dry <wink>.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
39
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
40 The primary thing it missed is that when a weakref to a piece of cyclic
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
41 trash (CT) exists, then any call to any Python code whatsoever can end up
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
42 materializing a strong reference to that weakref's CT referent, and so
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
43 possibly resurrect an insane object (one for which cyclic gc has called-- or
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
44 will call before it's done --tp_clear()). It's not even necessarily that a
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
45 weakref callback or __del__ method does something nasty on purpose: as
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
46 soon as we execute Python code, threads other than the gc thread can run
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
47 too, and they can do ordinary things with weakrefs that end up resurrecting
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
48 CT while gc is running.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
49
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
50 http://www.python.org/sf/1055820
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
51
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
52 shows how innocent it can be, and also how nasty. Variants of the three
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
53 focussed test cases attached to that bug report are now part of Python's
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
54 standard Lib/test/test_gc.py.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
55
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
56 Jim Fulton gave the best nutshell summary of the new (in 2.4 and 2.3.5)
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
57 approach:
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
58
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
59 Clearing cyclic trash can call Python code. If there are weakrefs to
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
60 any of the cyclic trash, then those weakrefs can be used to resurrect
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
61 the objects. Therefore, *before* clearing cyclic trash, we need to
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
62 remove any weakrefs. If any of the weakrefs being removed have
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
63 callbacks, then we need to save the callbacks and call them *after* all
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
64 of the weakrefs have been cleared.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
65
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
66 Alas, doing just that much doesn't work, because it overlooks what turned
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
67 out to be the much subtler problems that were fixed earlier, and described
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
68 below. We do clear all weakrefs to CT now before breaking cycles, but not
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
69 all callbacks encountered can be run later. That's explained in horrid
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
70 detail below.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
71
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
72 Older text follows, with a some later comments in [] brackets:
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
73
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
74 Before 2.3.3
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
75 ============
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
76
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
77 Before 2.3.3, Python's cyclic gc didn't pay any attention to weakrefs.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
78 Segfaults in Zope3 resulted.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
79
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
80 weakrefs in Python are designed to, at worst, let *other* objects learn
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
81 that a given object has died, via a callback function. The weakly
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
82 referenced object itself is not passed to the callback, and the presumption
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
83 is that the weakly referenced object is unreachable trash at the time the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
84 callback is invoked.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
85
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
86 That's usually true, but not always. Suppose a weakly referenced object
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
87 becomes part of a clump of cyclic trash. When enough cycles are broken by
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
88 cyclic gc that the object is reclaimed, the callback is invoked. If it's
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
89 possible for the callback to get at objects in the cycle(s), then it may be
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
90 possible for those objects to access (via strong references in the cycle)
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
91 the weakly referenced object being torn down, or other objects in the cycle
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
92 that have already suffered a tp_clear() call. There's no guarantee that an
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
93 object is in a sane state after tp_clear(). Bad things (including
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
94 segfaults) can happen right then, during the callback's execution, or can
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
95 happen at any later time if the callback manages to resurrect an insane
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
96 object.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
97
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
98 [That missed that, in addition, a weakref to CT can exist outside CT, and
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
99 any callback into Python can use such a non-CT weakref to resurrect its CT
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
100 referent. The same bad kinds of things can happen then.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
101
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
102 Note that if it's possible for the callback to get at objects in the trash
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
103 cycles, it must also be the case that the callback itself is part of the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
104 trash cycles. Else the callback would have acted as an external root to
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
105 the current collection, and nothing reachable from it would be in cyclic
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
106 trash either.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
107
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
108 [Except that a non-CT callback can also use a non-CT weakref to get at
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
109 CT objects.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
110
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
111 More, if the callback itself is in cyclic trash, then the weakref to which
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
112 the callback is attached must also be trash, and for the same kind of
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
113 reason: if the weakref acted as an external root, then the callback could
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
114 not have been cyclic trash.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
115
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
116 So a problem here requires that a weakref, that weakref's callback, and the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
117 weakly referenced object, all be in cyclic trash at the same time. This
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
118 isn't easy to stumble into by accident while Python is running, and, indeed,
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
119 it took quite a while to dream up failing test cases. Zope3 saw segfaults
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
120 during shutdown, during the second call of gc in Py_Finalize, after most
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
121 modules had been torn down. That creates many trash cycles (esp. those
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
122 involving new-style classes), making the problem much more likely. Once you
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
123 know what's required to provoke the problem, though, it's easy to create
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
124 tests that segfault before shutdown.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
125
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
126 In 2.3.3, before breaking cycles, we first clear all the weakrefs with
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
127 callbacks in cyclic trash. Since the weakrefs *are* trash, and there's no
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
128 defined-- or even predictable --order in which tp_clear() gets called on
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
129 cyclic trash, it's defensible to first clear weakrefs with callbacks. It's
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
130 a feature of Python's weakrefs too that when a weakref goes away, the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
131 callback (if any) associated with it is thrown away too, unexecuted.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
132
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
133 [In 2.4/2.3.5, we first clear all weakrefs to CT objects, whether or not
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
134 those weakrefs are themselves CT, and whether or not they have callbacks.
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
135 The callbacks (if any) on non-CT weakrefs (if any) are invoked later,
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
136 after all weakrefs-to-CT have been cleared. The callbacks (if any) on CT
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
137 weakrefs (if any) are never invoked, for the excruciating reasons
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
138 explained here.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
139
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
140 Just that much is almost enough to prevent problems, by throwing away
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
141 *almost* all the weakref callbacks that could get triggered by gc. The
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
142 problem remaining is that clearing a weakref with a callback decrefs the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
143 callback object, and the callback object may *itself* be weakly referenced,
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
144 via another weakref with another callback. So the process of clearing
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
145 weakrefs can trigger callbacks attached to other weakrefs, and those
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
146 latter weakrefs may or may not be part of cyclic trash.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
147
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
148 So, to prevent any Python code from running while gc is invoking tp_clear()
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
149 on all the objects in cyclic trash,
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
150
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
151 [That was always wrong: we can't stop Python code from running when gc
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
152 is breaking cycles. If an object with a __del__ method is not itself in
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
153 a cycle, but is reachable only from CT, then breaking cycles will, as a
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
154 matter of course, drop the refcount on that object to 0, and its __del__
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
155 will run right then. What we can and must stop is running any Python
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
156 code that could access CT.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
157 it's not quite enough just to invoke
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
158 tp_clear() on weakrefs with callbacks first. Instead the weakref module
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
159 grew a new private function (_PyWeakref_ClearRef) that does only part of
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
160 tp_clear(): it removes the weakref from the weakly-referenced object's list
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
161 of weakrefs, but does not decref the callback object. So calling
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
162 _PyWeakref_ClearRef(wr) ensures that wr's callback object will never
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
163 trigger, and (unlike weakref's tp_clear()) also prevents any callback
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
164 associated *with* wr's callback object from triggering.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
165
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
166 [Although we may trigger such callbacks later, as explained below.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
167
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
168 Then we can call tp_clear on all the cyclic objects and never trigger
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
169 Python code.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
170
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
171 [As above, not so: it means never trigger Python code that can access CT.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
172
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
173 After we do that, the callback objects still need to be decref'ed. Callbacks
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
174 (if any) *on* the callback objects that were also part of cyclic trash won't
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
175 get invoked, because we cleared all trash weakrefs with callbacks at the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
176 start. Callbacks on the callback objects that were not part of cyclic trash
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
177 acted as external roots to everything reachable from them, so nothing
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
178 reachable from them was part of cyclic trash, so gc didn't do any damage to
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
179 objects reachable from them, and it's safe to call them at the end of gc.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
180
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
181 [That's so. In addition, now we also invoke (if any) the callbacks on
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
182 non-CT weakrefs to CT objects, during the same pass that decrefs the
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
183 callback objects.]
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
184
30990
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
185 An alternative would have been to treat objects with callbacks like objects
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
186 with __del__ methods, refusing to collect them, appending them to gc.garbage
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
187 instead. That would have been much easier. Jim Fulton gave a strong
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
188 argument against that (on Python-Dev):
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
189
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
190 There's a big difference between __del__ and weakref callbacks.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
191 The __del__ method is "internal" to a design. When you design a
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
192 class with a del method, you know you have to avoid including the
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
193 class in cycles.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
194
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
195 Now, suppose you have a design that makes has no __del__ methods but
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
196 that does use cyclic data structures. You reason about the design,
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
197 run tests, and convince yourself you don't have a leak.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
198
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
199 Now, suppose some external code creates a weakref to one of your
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
200 objects. All of a sudden, you start leaking. You can look at your
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
201 code all you want and you won't find a reason for the leak.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
202
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
203 IOW, a class designer can out-think __del__ problems, but has no control
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
204 over who creates weakrefs to his classes or class instances. The class
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
205 user has little chance either of predicting when the weakrefs he creates
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
206 may end up in cycles.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
207
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
208 Callbacks on weakref callbacks are executed in an arbitrary order, and
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
209 that's not good (a primary reason not to collect cycles with objects with
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
210 __del__ methods is to avoid running finalizers in an arbitrary order).
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
211 However, a weakref callback on a weakref callback has got to be rare.
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
212 It's possible to do such a thing, so gc has to be robust against it, but
8326f64244c2 This commit was manufactured by cvs2svn to create branch
cvs2svn <tools@python.org>
parents:
diff changeset
213 I doubt anyone has done it outside the test case I wrote for it.
33862
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
214
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
215 [The callbacks (if any) on non-CT weakrefs to CT objects are also executed
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
216 in an arbitrary order now. But they were before too, depending on the
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
217 vagaries of when tp_clear() happened to break enough cycles to trigger
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
218 them. People simply shouldn't try to use __del__ or weakref callbacks to
facb5a2f4b53 SF 1055820: weakref callback vs gc vs threads
Tim Peters <tim.peters@gmail.com>
parents: 30990
diff changeset
219 do fancy stuff.]