Mercurial > cpython-lru_cache-weakref

annotate Tools/scripts/db2pickle.py @ 87690:66c1c9f32567 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859
author Radomir Dopieralski <devel@sheep.art.pl>
date Tue, 03 Dec 2013 16:02:48 +0100
parents fcfe324663a6
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
59522
fcfe324663a6 convert shebang lines: python -> python3
Benjamin Peterson <benjamin@python.org>
parents: 57145
diff changeset
1 #!/usr/bin/env python3
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
2
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
3 """
30804
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
4 Synopsis: %(prog)s [-h|-g|-b|-r|-a] dbfile [ picklefile ]
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
5
29549
Skip Montanaro <skip@pobox.com>
parents: 28774
diff changeset
6 Convert the database file given on the command line to a pickle
31613
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
7 representation. The optional flags indicate the type of the database:
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
8
47285
afbde360dd1a Create the dbm package from PEP 3108. #2881.
Georg Brandl <georg@python.org>
parents: 31613
diff changeset
9 -a - open using dbm (any supported format)
31613
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
10 -b - open as bsddb btree file
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
11 -d - open as dbm file
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
12 -g - open as gdbm file
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
13 -h - open as bsddb hash file
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
14 -r - open as bsddb recno file
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
15
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
16 The default is hash. If a pickle file is named it is opened for write
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
17 access (deleting any existing data). If no pickle file is named, the pickle
46df08eb0839 * explain flags in doc strings
Skip Montanaro <skip@pobox.com>
parents: 30804
diff changeset
18 output is written to standard output.
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
19
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
20 """
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
21
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
22 import getopt
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
23 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
24 import bsddb
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
25 except ImportError:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
26 bsddb = None
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
27 try:
47285
afbde360dd1a Create the dbm package from PEP 3108. #2881.
Georg Brandl <georg@python.org>
parents: 31613
diff changeset
28 import dbm.ndbm as dbm
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
29 except ImportError:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
30 dbm = None
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
31 try:
47285
afbde360dd1a Create the dbm package from PEP 3108. #2881.
Georg Brandl <georg@python.org>
parents: 31613
diff changeset
32 import dbm.gnu as gdbm
30804
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
33 except ImportError:
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
34 gdbm = None
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
35 try:
57145
979eb1bbf6e6 Merged revisions 76156 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 47285
diff changeset
36 import dbm.ndbm as anydbm
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
37 except ImportError:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
38 anydbm = None
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
39 import sys
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
40 try:
57145
979eb1bbf6e6 Merged revisions 76156 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 47285
diff changeset
41 import pickle as pickle
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
42 except ImportError:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
43 import pickle
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
44
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
45 prog = sys.argv[0]
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
46
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
47 def usage():
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
48 sys.stderr.write(__doc__ % globals())
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
49
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
50 def main(args):
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
51 try:
30804
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
52 opts, args = getopt.getopt(args, "hbrdag",
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
53 ["hash", "btree", "recno", "dbm",
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
54 "gdbm", "anydbm"])
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
55 except getopt.error:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
56 usage()
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
57 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
58
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
59 if len(args) == 0 or len(args) > 2:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
60 usage()
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
61 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
62 elif len(args) == 1:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
63 dbfile = args[0]
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
64 pfile = sys.stdout
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
65 else:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
66 dbfile = args[0]
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
67 try:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
68 pfile = open(args[1], 'wb')
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
69 except IOError:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
70 sys.stderr.write("Unable to open %s\n" % args[1])
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
71 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
72
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
73 dbopen = None
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
74 for opt, arg in opts:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
75 if opt in ("-h", "--hash"):
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
76 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
77 dbopen = bsddb.hashopen
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
78 except AttributeError:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
79 sys.stderr.write("bsddb module unavailable.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
80 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
81 elif opt in ("-b", "--btree"):
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
82 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
83 dbopen = bsddb.btopen
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
84 except AttributeError:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
85 sys.stderr.write("bsddb module unavailable.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
86 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
87 elif opt in ("-r", "--recno"):
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
88 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
89 dbopen = bsddb.rnopen
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
90 except AttributeError:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
91 sys.stderr.write("bsddb module unavailable.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
92 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
93 elif opt in ("-a", "--anydbm"):
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
94 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
95 dbopen = anydbm.open
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
96 except AttributeError:
47285
afbde360dd1a Create the dbm package from PEP 3108. #2881.
Georg Brandl <georg@python.org>
parents: 31613
diff changeset
97 sys.stderr.write("dbm module unavailable.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
98 return 1
30804
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
99 elif opt in ("-g", "--gdbm"):
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
100 try:
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
101 dbopen = gdbm.open
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
102 except AttributeError:
47285
afbde360dd1a Create the dbm package from PEP 3108. #2881.
Georg Brandl <georg@python.org>
parents: 31613
diff changeset
103 sys.stderr.write("dbm.gnu module unavailable.\n")
30804
ac8c7716a85d allow dump/load of gdbm files
Skip Montanaro <skip@pobox.com>
parents: 29549
diff changeset
104 return 1
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
105 elif opt in ("-d", "--dbm"):
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
106 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
107 dbopen = dbm.open
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
108 except AttributeError:
47285
afbde360dd1a Create the dbm package from PEP 3108. #2881.
Georg Brandl <georg@python.org>
parents: 31613
diff changeset
109 sys.stderr.write("dbm.ndbm module unavailable.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
110 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
111 if dbopen is None:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
112 if bsddb is None:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
113 sys.stderr.write("bsddb module unavailable - ")
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
114 sys.stderr.write("must specify dbtype.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
115 return 1
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
116 else:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
117 dbopen = bsddb.hashopen
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
118
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
119 try:
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
120 db = dbopen(dbfile, 'r')
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
121 except bsddb.error:
27641
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
122 sys.stderr.write("Unable to open %s. " % dbfile)
dd73a1218f69 try to avoid gross incompatibilities with older versions of Python
Skip Montanaro <skip@pobox.com>
parents: 27632
diff changeset
123 sys.stderr.write("Check for format or version mismatch.\n")
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
124 return 1
28774
c9bee11eaa8b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 27641
diff changeset
125
27632
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
126 for k in db.keys():
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
127 pickle.dump((k, db[k]), pfile, 1==1)
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
128
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
129 db.close()
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
130 pfile.close()
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
131
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
132 return 0
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
133
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
134 if __name__ == "__main__":
640e52dee1a0 dump database files to pickle format
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
135 sys.exit(main(sys.argv[1:]))