cpython-lru_cache-weakref: Tools/scripts/combinerefs.py annotate

annotate Tools/scripts/combinerefs.py @ 80661:94de77bd0b4b 2.7

Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.

author	Andrew Svetlov <andrew.svetlov@gmail.com>
date	Thu, 29 Nov 2012 19:51:28 +0200
parents	018170a4a2d7
children	46625e6122e9

rev	line source
28669 0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	1 #! /usr/bin/env python
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	2
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	3 """
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	4 combinerefs path
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	5
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	6 A helper for analyzing PYTHONDUMPREFS output.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	7
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	8 When the PYTHONDUMPREFS envar is set in a debug build, at Python shutdown
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	9 time Py_Finalize() prints the list of all live objects twice: first it
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	10 prints the repr() of each object while the interpreter is still fully intact.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	11 After cleaning up everything it can, it prints all remaining live objects
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	12 again, but the second time just prints their addresses, refcounts, and type
28697 018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	13 names (because the interpreter has been torn down, calling repr methods at
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	14 this point can get into infinite loops or blow up).
28669 0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	15
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	16 Save all this output into a file, then run this script passing the path to
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	17 that file. The script finds both output chunks, combines them, then prints
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	18 a line of output for each object still alive at the end:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	19
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	20 address refcnt typename repr
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	21
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	22 address is the address of the object, in whatever format the platform C
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	23 produces for a %p format code.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	24
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	25 refcnt is of the form
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	26
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	27 "[" ref "]"
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	28
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	29 when the object's refcount is the same in both PYTHONDUMPREFS output blocks,
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	30 or
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	31
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	32 "[" ref_before "->" ref_after "]"
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	33
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	34 if the refcount changed.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	35
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	36 typename is object->ob_type->tp_name, extracted from the second PYTHONDUMPREFS
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	37 output block.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	38
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	39 repr is repr(object), extracted from the first PYTHONDUMPREFS output block.
28697 018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	40 CAUTION: If object is a container type, it may not actually contain all the
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	41 objects shown in the repr: the repr was captured from the first output block,
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	42 and some of the containees may have been released since then. For example,
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	43 it's common for the line showing the dict of interned strings to display
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	44 strings that no longer exist at the end of Py_Finalize; this can be recognized
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	45 (albeit painfully) because such containees don't have a line of their own.
28669 0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	46
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	47 The objects are listed in allocation order, with most-recently allocated
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	48 printed first, and the first object allocated printed last.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	49
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	50
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	51 Simple examples:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	52
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	53 00857060 [14] str '__len__'
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	54
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	55 The str object '__len__' is alive at shutdown time, and both PYTHONDUMPREFS
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	56 output blocks said there were 14 references to it. This is probably due to
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	57 C modules that intern the string "__len__" and keep a reference to it in a
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	58 file static.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	59
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	60 00857038 [46->5] tuple ()
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	61
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	62 46-5 = 41 references to the empty tuple were removed by the cleanup actions
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	63 between the times PYTHONDUMPREFS produced output.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	64
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	65 00858028 [1025->1456] str '<dummy key>'
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	66
28697 018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	67 The string '<dummy key>', which is used in dictobject.c to overwrite a real
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	68 key that gets deleted, grew several hundred references during cleanup. It
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	69 suggests that stuff did get removed from dicts by cleanup, but that the dicts
018170a4a2d7 Docstring clarifications. Tim Peters <tim.peters@gmail.com> parents: 28677 diff changeset	70 themselves are staying alive for some reason. """
28669 0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	71
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	72 import re
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	73 import sys
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	74
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	75 # Generate lines from fileiter. If whilematch is true, continue reading
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	76 # while the regexp object pat matches line. If whilematch is false, lines
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	77 # are read so long as pat doesn't match them. In any case, the first line
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	78 # that doesn't match pat (when whilematch is true), or that does match pat
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	79 # (when whilematch is false), is lost, and fileiter will resume at the line
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	80 # following it.
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	81 def read(fileiter, pat, whilematch):
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	82 for line in fileiter:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	83 if bool(pat.match(line)) == whilematch:
28670 c555cd02b0f9 Heh -- I checked in an old version of this, at a time I documented the Tim Peters <tim.peters@gmail.com> parents: 28669 diff changeset	84 yield line
28669 0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	85 else:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	86 break
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	87
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	88 def combine(fname):
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	89 f = file(fname)
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	90 fi = iter(f)
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	91
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	92 for line in read(fi, re.compile(r'^Remaining objects:$'), False):
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	93 pass
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	94
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	95 crack = re.compile(r'([a-zA-Z\d]+) \[(\d+)\] (.*)')
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	96 addr2rc = {}
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	97 addr2guts = {}
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	98 before = 0
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	99 for line in read(fi, re.compile(r'^Remaining object addresses:$'), False):
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	100 m = crack.match(line)
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	101 if m:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	102 addr, addr2rc[addr], addr2guts[addr] = m.groups()
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	103 before += 1
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	104 else:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	105 print '??? skipped:', line
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	106
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	107 after = 0
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	108 for line in read(fi, crack, True):
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	109 after += 1
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	110 m = crack.match(line)
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	111 assert m
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	112 addr, rc, guts = m.groups() # guts is type name here
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	113 if addr not in addr2rc:
28677 1815166e8097 Don't double-space the "??? new object ..." messages. Guido van Rossum <guido@python.org> parents: 28670 diff changeset	114 print '??? new object created while tearing down:', line.rstrip()
28669 0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	115 continue
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	116 print addr,
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	117 if rc == addr2rc[addr]:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	118 print '[%s]' % rc,
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	119 else:
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	120 print '[%s->%s]' % (addr2rc[addr], rc),
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	121 print guts, addr2guts[addr]
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	122
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	123 f.close()
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	124 print "%d objects before, %d after" % (before, after)
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	125
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	126 if __name__ == '__main__':
0d8d4c8dc68b _Py_PrintReferenceAddresses(): also print the type name. In real use Tim Peters <tim.peters@gmail.com> parents: diff changeset	127 combine(sys.argv[1])

Mercurial > cpython-lru_cache-weakref

annotate Tools/scripts/combinerefs.py @ 80661:94de77bd0b4b 2.7