cpython-lru_cache-weakref: Lib/linecache.py annotate

annotate Lib/linecache.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	7d69d04522e3
children

rev	line source
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	1 """Cache lines from files.
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	2
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	3 This is intended to read lines from modules imported -- hence if a filename
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	4 is not found, it will look down the module search path for a file by
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	5 that name.
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	6 """
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	7
722 57e94eba2484 Search through the module search path. Guido van Rossum <guido@python.org> parents: 566 diff changeset	8 import sys
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	9 import os
52325 fdc806c77cd6 reuse tokenize.detect_encoding in linecache instead of a custom solution Benjamin Peterson <benjamin@python.org> parents: 50163 diff changeset	10 import tokenize
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	11
29680 4e1a47f5204c Whitespace normalization. Jeremy Hylton <jeremy@alum.mit.edu> parents: 23625 diff changeset	12 __all__ = ["getline", "clearcache", "checkcache"]
16198 32755b456369 a few more modules get __all__ Skip Montanaro <skip@pobox.com> parents: 15543 diff changeset	13
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	14 def getline(filename, lineno, module_globals=None):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	15 lines = getlines(filename, module_globals)
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	16 if 1 <= lineno <= len(lines):
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	17 return lines[lineno-1]
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	18 else:
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	19 return ''
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	20
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	21
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	22 # The cache
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	23
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	24 cache = {} # The cache
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	25
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	26
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	27 def clearcache():
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	28 """Clear the cache entirely."""
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	29
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	30 global cache
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	31 cache = {}
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	32
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	33
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	34 def getlines(filename, module_globals=None):
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	35 """Get the lines for a file from the cache.
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	36 Update the cache if it doesn't contain an entry for this file already."""
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	37
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23029 diff changeset	38 if filename in cache:
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	39 return cache[filename][2]
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	40 else:
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	41 return updatecache(filename, module_globals)
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	42
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	43
33839 4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	44 def checkcache(filename=None):
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	45 """Discard cache entries that are out of date.
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	46 (This is not checked upon each call!)"""
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	47
33839 4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	48 if filename is None:
40948 3093ea9b4024 Fix another loop over a dict that may change... :-( Guido van Rossum <guido@python.org> parents: 40587 diff changeset	49 filenames = list(cache.keys())
33839 4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	50 else:
4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	51 if filename in cache:
4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	52 filenames = [filename]
4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	53 else:
4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	54 return
4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	55
4dd97c37c2cc SF #737473: Show up-to-date source code in tracebacks always. Hye-Shik Chang <hyeshik@gmail.com> parents: 29680 diff changeset	56 for filename in filenames:
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	57 size, mtime, lines, fullname = cache[filename]
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	58 if mtime is None:
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	59 continue # no-op for files loaded via a __loader__
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	60 try:
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	61 stat = os.stat(fullname)
80932 47f98a550d42 Issue #16706: get rid of os.error Andrew Svetlov <andrew.svetlov@gmail.com> parents: 66097 diff changeset	62 except OSError:
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	63 del cache[filename]
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	64 continue
23625 65cf478950ff Replaced obsolete stat module constants with equivalent attributes Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	65 if size != stat.st_size or mtime != stat.st_mtime:
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	66 del cache[filename]
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	67
5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	68
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	69 def updatecache(filename, module_globals=None):
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	70 """Update a cache entry and return its list of lines.
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	71 If something's wrong, print a message, discard the cache entry,
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	72 and return an empty list."""
566 5a1f9ade1512 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	73
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23029 diff changeset	74 if filename in cache:
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	75 del cache[filename]
61828 f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	76 if not filename or (filename.startswith('<') and filename.endswith('>')):
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	77 return []
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	78
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	79 fullname = filename
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	80 try:
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	81 stat = os.stat(fullname)
61828 f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	82 except OSError:
53775 b12a44eef40b Merged revisions 72319-72320 via svnmerge from Georg Brandl <georg@python.org> parents: 52325 diff changeset	83 basename = filename
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	84
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	85 # Try for a __loader__, if available
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	86 if module_globals and '__loader__' in module_globals:
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	87 name = module_globals.get('__name__')
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	88 loader = module_globals['__loader__']
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	89 get_source = getattr(loader, 'get_source', None)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	90
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	91 if name and get_source:
50163 dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	92 try:
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	93 data = get_source(name)
81039 7d69d04522e3 Replace IOError with OSError (#16715) Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80932 diff changeset	94 except (ImportError, OSError):
50163 dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	95 pass
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	96 else:
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	97 if data is None:
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	98 # No luck, the PEP302 loader cannot find the source
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	99 # for this module.
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	100 return []
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	101 cache[filename] = (
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	102 len(data), None,
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	103 [line+'\n' for line in data.splitlines()], fullname
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	104 )
dd1b8da01d02 Merged revisions 67750-67751 via svnmerge from Nick Coghlan <ncoghlan@gmail.com> parents: 50128 diff changeset	105 return cache[filename][2]
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	106
53775 b12a44eef40b Merged revisions 72319-72320 via svnmerge from Georg Brandl <georg@python.org> parents: 52325 diff changeset	107 # Try looking through the module search path, which is only useful
b12a44eef40b Merged revisions 72319-72320 via svnmerge from Georg Brandl <georg@python.org> parents: 52325 diff changeset	108 # when handling a relative filename.
b12a44eef40b Merged revisions 72319-72320 via svnmerge from Georg Brandl <georg@python.org> parents: 52325 diff changeset	109 if os.path.isabs(filename):
b12a44eef40b Merged revisions 72319-72320 via svnmerge from Georg Brandl <georg@python.org> parents: 52325 diff changeset	110 return []
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 33839 diff changeset	111
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	112 for dirname in sys.path:
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	113 try:
17977 2a94bf235daa Patch from Gordon McMillan. Tim Peters <tim.peters@gmail.com> parents: 16198 diff changeset	114 fullname = os.path.join(dirname, basename)
2a94bf235daa Patch from Gordon McMillan. Tim Peters <tim.peters@gmail.com> parents: 16198 diff changeset	115 except (TypeError, AttributeError):
2a94bf235daa Patch from Gordon McMillan. Tim Peters <tim.peters@gmail.com> parents: 16198 diff changeset	116 # Not sufficiently string-like to do anything useful with.
61828 f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	117 continue
f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	118 try:
f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	119 stat = os.stat(fullname)
f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	120 break
80932 47f98a550d42 Issue #16706: get rid of os.error Andrew Svetlov <andrew.svetlov@gmail.com> parents: 66097 diff changeset	121 except OSError:
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	122 pass
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	123 else:
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	124 return []
61828 f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	125 try:
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 65004 diff changeset	126 with tokenize.open(fullname) as fp:
61828 f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	127 lines = fp.readlines()
81039 7d69d04522e3 Replace IOError with OSError (#16715) Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80932 diff changeset	128 except OSError:
65004 0ee2113a4fde linecache.updatecache(): don't the lines into the cache on IOError Victor Stinner <victor.stinner@haypocalc.com> parents: 65002 diff changeset	129 return []
61828 f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	130 if lines and not lines[-1].endswith('\n'):
f389563ff372 Merged revisions 81428-81429,81432-81433,81437 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53775 diff changeset	131 lines[-1] += '\n'
23625 65cf478950ff Replaced obsolete stat module constants with equivalent attributes Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	132 size, mtime = stat.st_size, stat.st_mtime
11706 b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	133 cache[filename] = size, mtime, lines, fullname
b6f52c1ef6da More trivial comment -> docstring transformations by Ka-Ping Yee, Guido van Rossum <guido@python.org> parents: 2954 diff changeset	134 return lines

Mercurial > cpython-lru_cache-weakref

annotate Lib/linecache.py @ 87690:66c1c9f32567 default tip