Mercurial > cpython-lru_cache-weakref

annotate Lib/multiprocessing/pool.py @ 79675:c0a2294de787 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16199: Backport 2to3 typo fix from 3.2.
author Chris Jerdonek <chris.jerdonek@gmail.com>
date Thu, 11 Oct 2012 16:15:46 -0700
parents 4c07b9c49b75
children 2684176519ef
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
1 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
2 # Module providing the `Pool` class for managing a process pool
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
3 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
4 # multiprocessing/pool.py
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
5 #
66887
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
6 # Copyright (c) 2006-2008, R Oudkerk
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
7 # All rights reserved.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
8 #
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
9 # Redistribution and use in source and binary forms, with or without
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
10 # modification, are permitted provided that the following conditions
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
11 # are met:
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
12 #
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
13 # 1. Redistributions of source code must retain the above copyright
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
14 # notice, this list of conditions and the following disclaimer.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
15 # 2. Redistributions in binary form must reproduce the above copyright
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
16 # notice, this list of conditions and the following disclaimer in the
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
17 # documentation and/or other materials provided with the distribution.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
18 # 3. Neither the name of author nor the names of any contributors may be
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
19 # used to endorse or promote products derived from this software
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
20 # without specific prior written permission.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
21 #
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
22 # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
23 # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
24 # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
25 # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
26 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
27 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
28 # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
29 # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
30 # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
31 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 59464
diff changeset
32 # SUCH DAMAGE.
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
33 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
34
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
35 __all__ = ['Pool']
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
36
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
37 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
38 # Imports
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
39 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
40
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
41 import threading
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
42 import Queue
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
43 import itertools
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
44 import collections
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
45 import time
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
46
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
47 from multiprocessing import Process, cpu_count, TimeoutError
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
48 from multiprocessing.util import Finalize, debug
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
49
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
50 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
51 # Constants representing the state of a pool
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
52 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
53
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
54 RUN = 0
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
55 CLOSE = 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
56 TERMINATE = 2
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
57
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
58 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
59 # Miscellaneous
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
60 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
61
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
62 job_counter = itertools.count()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
63
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
64 def mapstar(args):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
65 return map(*args)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
66
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
67 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
68 # Code run by worker processes
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
69 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
70
76712
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
71 class MaybeEncodingError(Exception):
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
72 """Wraps possible unpickleable errors, so they can be
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
73 safely sent through the socket."""
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
74
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
75 def __init__(self, exc, value):
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
76 self.exc = repr(exc)
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
77 self.value = repr(value)
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
78 super(MaybeEncodingError, self).__init__(self.exc, self.value)
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
79
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
80 def __str__(self):
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
81 return "Error sending result: '%s'. Reason: '%s'" % (self.value,
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
82 self.exc)
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
83
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
84 def __repr__(self):
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
85 return "<MaybeEncodingError: %s>" % str(self)
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
86
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
87
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
88 def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None):
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
89 assert maxtasks is None or (type(maxtasks) == int and maxtasks > 0)
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
90 put = outqueue.put
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
91 get = inqueue.get
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
92 if hasattr(inqueue, '_writer'):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
93 inqueue._writer.close()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
94 outqueue._reader.close()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
95
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
96 if initializer is not None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
97 initializer(*initargs)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
98
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
99 completed = 0
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
100 while maxtasks is None or (maxtasks and completed < maxtasks):
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
101 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
102 task = get()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
103 except (EOFError, IOError):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
104 debug('worker got EOFError or IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
105 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
106
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
107 if task is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
108 debug('worker got sentinel -- exiting')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
109 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
110
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
111 job, i, func, args, kwds = task
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
112 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
113 result = (True, func(*args, **kwds))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
114 except Exception, e:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
115 result = (False, e)
76712
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
116 try:
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
117 put((job, i, result))
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
118 except Exception as e:
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
119 wrapped = MaybeEncodingError(e, result[1])
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
120 debug("Possible encoding error while sending result: %s" % (
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
121 wrapped))
26bbff4562a7 Issue #9400: Partial backport of fix for #9244
Richard Oudkerk <shibturn@gmail.com>
parents: 73099
diff changeset
122 put((job, i, (False, wrapped)))
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
123 completed += 1
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
124 debug('worker exiting after %d tasks' % completed)
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
125
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
126 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
127 # Class representing a process pool
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
128 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
129
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
130 class Pool(object):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
131 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
132 Class which supports an async version of the `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
133 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
134 Process = Process
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
135
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
136 def __init__(self, processes=None, initializer=None, initargs=(),
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
137 maxtasksperchild=None):
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
138 self._setup_queues()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
139 self._taskqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
140 self._cache = {}
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
141 self._state = RUN
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
142 self._maxtasksperchild = maxtasksperchild
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
143 self._initializer = initializer
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
144 self._initargs = initargs
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
145
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
146 if processes is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
147 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
148 processes = cpu_count()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
149 except NotImplementedError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
150 processes = 1
70911
815263546757 Issue #12285: multiprocessing.Pool() raises a ValueError if the number of
Victor Stinner <victor.stinner@haypocalc.com>
parents: 69239
diff changeset
151 if processes < 1:
815263546757 Issue #12285: multiprocessing.Pool() raises a ValueError if the number of
Victor Stinner <victor.stinner@haypocalc.com>
parents: 69239
diff changeset
152 raise ValueError("Number of processes must be at least 1")
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
153
52727
71f6129ddad4 Add custom initializer argument to multiprocess.Manager*, courtesy of lekma
Jesse Noller <jnoller@gmail.com>
parents: 50002
diff changeset
154 if initializer is not None and not hasattr(initializer, '__call__'):
71f6129ddad4 Add custom initializer argument to multiprocess.Manager*, courtesy of lekma
Jesse Noller <jnoller@gmail.com>
parents: 50002
diff changeset
155 raise TypeError('initializer must be a callable')
71f6129ddad4 Add custom initializer argument to multiprocess.Manager*, courtesy of lekma
Jesse Noller <jnoller@gmail.com>
parents: 50002
diff changeset
156
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
157 self._processes = processes
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
158 self._pool = []
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
159 self._repopulate_pool()
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
160
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
161 self._worker_handler = threading.Thread(
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
162 target=Pool._handle_workers,
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
163 args=(self, )
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
164 )
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
165 self._worker_handler.daemon = True
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
166 self._worker_handler._state = RUN
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
167 self._worker_handler.start()
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
168
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
169
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
170 self._task_handler = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
171 target=Pool._handle_tasks,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
172 args=(self._taskqueue, self._quick_put, self._outqueue, self._pool)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
173 )
48291
28d93c9c3a03 patch up multiprocessing until it's API can be changed too
Benjamin Peterson <benjamin@python.org>
parents: 47696
diff changeset
174 self._task_handler.daemon = True
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
175 self._task_handler._state = RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
176 self._task_handler.start()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
177
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
178 self._result_handler = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
179 target=Pool._handle_results,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
180 args=(self._outqueue, self._quick_get, self._cache)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
181 )
48291
28d93c9c3a03 patch up multiprocessing until it's API can be changed too
Benjamin Peterson <benjamin@python.org>
parents: 47696
diff changeset
182 self._result_handler.daemon = True
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
183 self._result_handler._state = RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
184 self._result_handler.start()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
185
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
186 self._terminate = Finalize(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
187 self, self._terminate_pool,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
188 args=(self._taskqueue, self._inqueue, self._outqueue, self._pool,
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
189 self._worker_handler, self._task_handler,
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
190 self._result_handler, self._cache),
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
191 exitpriority=15
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
192 )
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
193
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
194 def _join_exited_workers(self):
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
195 """Cleanup after any worker processes which have exited due to reaching
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
196 their specified lifetime. Returns True if any workers were cleaned up.
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
197 """
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
198 cleaned = False
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
199 for i in reversed(range(len(self._pool))):
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
200 worker = self._pool[i]
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
201 if worker.exitcode is not None:
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
202 # worker exited
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
203 debug('cleaning up worker %d' % i)
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
204 worker.join()
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
205 cleaned = True
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
206 del self._pool[i]
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
207 return cleaned
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
208
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
209 def _repopulate_pool(self):
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
210 """Bring the number of pool processes up to the specified number,
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
211 for use after reaping workers which have exited.
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
212 """
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
213 for i in range(self._processes - len(self._pool)):
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
214 w = self.Process(target=worker,
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
215 args=(self._inqueue, self._outqueue,
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
216 self._initializer,
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
217 self._initargs, self._maxtasksperchild)
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
218 )
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
219 self._pool.append(w)
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
220 w.name = w.name.replace('Process', 'PoolWorker')
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
221 w.daemon = True
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
222 w.start()
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
223 debug('added worker')
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
224
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
225 def _maintain_pool(self):
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
226 """Clean up any exited workers and start replacements for them.
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
227 """
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
228 if self._join_exited_workers():
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
229 self._repopulate_pool()
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
230
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
231 def _setup_queues(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
232 from .queues import SimpleQueue
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
233 self._inqueue = SimpleQueue()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
234 self._outqueue = SimpleQueue()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
235 self._quick_put = self._inqueue._writer.send
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
236 self._quick_get = self._outqueue._reader.recv
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
237
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
238 def apply(self, func, args=(), kwds={}):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
239 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
240 Equivalent of `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
241 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
242 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
243 return self.apply_async(func, args, kwds).get()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
244
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
245 def map(self, func, iterable, chunksize=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
246 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
247 Equivalent of `map()` builtin
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
248 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
249 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
250 return self.map_async(func, iterable, chunksize).get()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
251
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
252 def imap(self, func, iterable, chunksize=1):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
253 '''
50002
5ef8b5a74efa Merged revisions 67245,67277,67289,67295,67301-67303,67307,67330,67332,67336,67355,67359,67362,67364,67367-67368,67370 via svnmerge from
Georg Brandl <georg@python.org>
parents: 48302
diff changeset
254 Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()`
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
255 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
256 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
257 if chunksize == 1:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
258 result = IMapIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
259 self._taskqueue.put((((result._job, i, func, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
260 for i, x in enumerate(iterable)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
261 return result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
262 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
263 assert chunksize > 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
264 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
265 result = IMapIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
266 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
267 for i, x in enumerate(task_batches)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
268 return (item for chunk in result for item in chunk)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
269
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
270 def imap_unordered(self, func, iterable, chunksize=1):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
271 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
272 Like `imap()` method but ordering of results is arbitrary
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
273 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
274 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
275 if chunksize == 1:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
276 result = IMapUnorderedIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
277 self._taskqueue.put((((result._job, i, func, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
278 for i, x in enumerate(iterable)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
279 return result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
280 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
281 assert chunksize > 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
282 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
283 result = IMapUnorderedIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
284 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
285 for i, x in enumerate(task_batches)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
286 return (item for chunk in result for item in chunk)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
287
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
288 def apply_async(self, func, args=(), kwds={}, callback=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
289 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
290 Asynchronous equivalent of `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
291 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
292 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
293 result = ApplyResult(self._cache, callback)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
294 self._taskqueue.put(([(result._job, None, func, args, kwds)], None))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
295 return result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
296
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
297 def map_async(self, func, iterable, chunksize=None, callback=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
298 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
299 Asynchronous equivalent of `map()` builtin
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
300 '''
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
301 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
302 if not hasattr(iterable, '__len__'):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
303 iterable = list(iterable)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
304
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
305 if chunksize is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
306 chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
307 if extra:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
308 chunksize += 1
55269
3370d608e5aa Issue 6433: multiprocessing.pool.map hangs on empty list
Jesse Noller <jnoller@gmail.com>
parents: 52727
diff changeset
309 if len(iterable) == 0:
3370d608e5aa Issue 6433: multiprocessing.pool.map hangs on empty list
Jesse Noller <jnoller@gmail.com>
parents: 52727
diff changeset
310 chunksize = 0
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
311
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
312 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
313 result = MapResult(self._cache, chunksize, len(iterable), callback)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
314 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
315 for i, x in enumerate(task_batches)), None))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
316 return result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
317
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
318 @staticmethod
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
319 def _handle_workers(pool):
73099
3465a9b2d25c Issue #10332: multiprocessing: fix a race condition when a Pool is closed
Charles-François Natali <neologix@free.fr>
parents: 70911
diff changeset
320 thread = threading.current_thread()
3465a9b2d25c Issue #10332: multiprocessing: fix a race condition when a Pool is closed
Charles-François Natali <neologix@free.fr>
parents: 70911
diff changeset
321
3465a9b2d25c Issue #10332: multiprocessing: fix a race condition when a Pool is closed
Charles-François Natali <neologix@free.fr>
parents: 70911
diff changeset
322 # Keep maintaining workers until the cache gets drained, unless the pool
3465a9b2d25c Issue #10332: multiprocessing: fix a race condition when a Pool is closed
Charles-François Natali <neologix@free.fr>
parents: 70911
diff changeset
323 # is terminated.
3465a9b2d25c Issue #10332: multiprocessing: fix a race condition when a Pool is closed
Charles-François Natali <neologix@free.fr>
parents: 70911
diff changeset
324 while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
325 pool._maintain_pool()
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
326 time.sleep(0.1)
69239
dfc61dc14f59 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
Antoine Pitrou <solipsis@pitrou.net>
parents: 66887
diff changeset
327 # send sentinel to stop workers
dfc61dc14f59 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
Antoine Pitrou <solipsis@pitrou.net>
parents: 66887
diff changeset
328 pool._taskqueue.put(None)
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
329 debug('worker handler exiting')
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
330
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
331 @staticmethod
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
332 def _handle_tasks(taskqueue, put, outqueue, pool):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
333 thread = threading.current_thread()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
334
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
335 for taskseq, set_length in iter(taskqueue.get, None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
336 i = -1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
337 for i, task in enumerate(taskseq):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
338 if thread._state:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
339 debug('task handler found thread._state != RUN')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
340 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
341 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
342 put(task)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
343 except IOError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
344 debug('could not put task on queue')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
345 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
346 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
347 if set_length:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
348 debug('doing set_length()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
349 set_length(i+1)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
350 continue
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
351 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
352 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
353 debug('task handler got sentinel')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
354
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
355
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
356 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
357 # tell result handler to finish when cache is empty
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
358 debug('task handler sending sentinel to result handler')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
359 outqueue.put(None)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
360
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
361 # tell workers there is no more work
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
362 debug('task handler sending sentinel to workers')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
363 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
364 put(None)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
365 except IOError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
366 debug('task handler got IOError when sending sentinels')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
367
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
368 debug('task handler exiting')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
369
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
370 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
371 def _handle_results(outqueue, get, cache):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
372 thread = threading.current_thread()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
373
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
374 while 1:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
375 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
376 task = get()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
377 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
378 debug('result handler got EOFError/IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
379 return
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
380
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
381 if thread._state:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
382 assert thread._state == TERMINATE
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
383 debug('result handler found thread._state=TERMINATE')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
384 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
385
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
386 if task is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
387 debug('result handler got sentinel')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
388 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
389
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
390 job, i, obj = task
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
391 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
392 cache[job]._set(i, obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
393 except KeyError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
394 pass
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
395
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
396 while cache and thread._state != TERMINATE:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
397 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
398 task = get()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
399 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
400 debug('result handler got EOFError/IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
401 return
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
402
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
403 if task is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
404 debug('result handler ignoring extra sentinel')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
405 continue
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
406 job, i, obj = task
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
407 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
408 cache[job]._set(i, obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
409 except KeyError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
410 pass
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
411
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
412 if hasattr(outqueue, '_reader'):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
413 debug('ensuring that outqueue is not full')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
414 # If we don't make room available in outqueue then
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
415 # attempts to add the sentinel (None) to outqueue may
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
416 # block. There is guaranteed to be no more than 2 sentinels.
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
417 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
418 for i in range(10):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
419 if not outqueue._reader.poll():
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
420 break
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
421 get()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
422 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
423 pass
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
424
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
425 debug('result handler exiting: len(cache)=%s, thread._state=%s',
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
426 len(cache), thread._state)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
427
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
428 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
429 def _get_tasks(func, it, size):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
430 it = iter(it)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
431 while 1:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
432 x = tuple(itertools.islice(it, size))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
433 if not x:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
434 return
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
435 yield (func, x)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
436
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
437 def __reduce__(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
438 raise NotImplementedError(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
439 'pool objects cannot be passed between processes or pickled'
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
440 )
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
441
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
442 def close(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
443 debug('closing pool')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
444 if self._state == RUN:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
445 self._state = CLOSE
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
446 self._worker_handler._state = CLOSE
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
447
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
448 def terminate(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
449 debug('terminating pool')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
450 self._state = TERMINATE
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
451 self._worker_handler._state = TERMINATE
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
452 self._terminate()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
453
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
454 def join(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
455 debug('joining pool')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
456 assert self._state in (CLOSE, TERMINATE)
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
457 self._worker_handler.join()
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
458 self._task_handler.join()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
459 self._result_handler.join()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
460 for p in self._pool:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
461 p.join()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
462
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
463 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
464 def _help_stuff_finish(inqueue, task_handler, size):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
465 # task_handler may be blocked trying to put items on inqueue
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
466 debug('removing tasks from inqueue until task handler finished')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
467 inqueue._rlock.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
468 while task_handler.is_alive() and inqueue._reader.poll():
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
469 inqueue._reader.recv()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
470 time.sleep(0)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
471
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
472 @classmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
473 def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
474 worker_handler, task_handler, result_handler, cache):
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
475 # this is guaranteed to only be called once
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
476 debug('finalizing pool')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
477
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
478 worker_handler._state = TERMINATE
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
479 task_handler._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
480
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
481 debug('helping task handler/workers to finish')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
482 cls._help_stuff_finish(inqueue, task_handler, len(pool))
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
483
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
484 assert result_handler.is_alive() or len(cache) == 0
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
485
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
486 result_handler._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
487 outqueue.put(None) # sentinel
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
488
69239
dfc61dc14f59 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
Antoine Pitrou <solipsis@pitrou.net>
parents: 66887
diff changeset
489 # We must wait for the worker handler to exit before terminating
dfc61dc14f59 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
Antoine Pitrou <solipsis@pitrou.net>
parents: 66887
diff changeset
490 # workers because we don't want workers to be restarted behind our back.
dfc61dc14f59 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
Antoine Pitrou <solipsis@pitrou.net>
parents: 66887
diff changeset
491 debug('joining worker handler')
77506
4c07b9c49b75 Issue #15101: Make pool finalizer avoid joining current thread
Richard Oudkerk <shibturn@gmail.com>
parents: 77377
diff changeset
492 if threading.current_thread() is not worker_handler:
4c07b9c49b75 Issue #15101: Make pool finalizer avoid joining current thread
Richard Oudkerk <shibturn@gmail.com>
parents: 77377
diff changeset
493 worker_handler.join(1e100)
69239
dfc61dc14f59 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
Antoine Pitrou <solipsis@pitrou.net>
parents: 66887
diff changeset
494
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
495 # Terminate workers which haven't already finished.
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
496 if pool and hasattr(pool[0], 'terminate'):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
497 debug('terminating workers')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
498 for p in pool:
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
499 if p.exitcode is None:
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
500 p.terminate()
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
501
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
502 debug('joining task handler')
77506
4c07b9c49b75 Issue #15101: Make pool finalizer avoid joining current thread
Richard Oudkerk <shibturn@gmail.com>
parents: 77377
diff changeset
503 if threading.current_thread() is not task_handler:
4c07b9c49b75 Issue #15101: Make pool finalizer avoid joining current thread
Richard Oudkerk <shibturn@gmail.com>
parents: 77377
diff changeset
504 task_handler.join(1e100)
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
505
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
506 debug('joining result handler')
77506
4c07b9c49b75 Issue #15101: Make pool finalizer avoid joining current thread
Richard Oudkerk <shibturn@gmail.com>
parents: 77377
diff changeset
507 if threading.current_thread() is not result_handler:
4c07b9c49b75 Issue #15101: Make pool finalizer avoid joining current thread
Richard Oudkerk <shibturn@gmail.com>
parents: 77377
diff changeset
508 result_handler.join(1e100)
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
509
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
510 if pool and hasattr(pool[0], 'terminate'):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
511 debug('joining pool workers')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
512 for p in pool:
59464
96e0011ea0a4 On finalize, don't try to join not started process.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59461
diff changeset
513 if p.is_alive():
58536
debe61673a8b Issue #6963: Added maxtasksperchild argument to multiprocessing.Pool
Jesse Noller <jnoller@gmail.com>
parents: 55269
diff changeset
514 # worker has not yet exited
59464
96e0011ea0a4 On finalize, don't try to join not started process.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59461
diff changeset
515 debug('cleaning up worker %d' % p.pid)
96e0011ea0a4 On finalize, don't try to join not started process.
Florent Xicluna <florent.xicluna@gmail.com>
parents: 59461
diff changeset
516 p.join()
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
517
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
518 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
519 # Class whose instances are returned by `Pool.apply_async()`
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
520 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
521
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
522 class ApplyResult(object):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
523
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
524 def __init__(self, cache, callback):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
525 self._cond = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
526 self._job = job_counter.next()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
527 self._cache = cache
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
528 self._ready = False
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
529 self._callback = callback
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
530 cache[self._job] = self
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
531
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
532 def ready(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
533 return self._ready
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
534
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
535 def successful(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
536 assert self._ready
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
537 return self._success
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
538
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
539 def wait(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
540 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
541 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
542 if not self._ready:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
543 self._cond.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
544 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
545 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
546
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
547 def get(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
548 self.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
549 if not self._ready:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
550 raise TimeoutError
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
551 if self._success:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
552 return self._value
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
553 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
554 raise self._value
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
555
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
556 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
557 self._success, self._value = obj
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
558 if self._callback and self._success:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
559 self._callback(self._value)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
560 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
561 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
562 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
563 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
564 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
565 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
566 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
567
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
568 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
569 # Class whose instances are returned by `Pool.map_async()`
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
570 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
571
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
572 class MapResult(ApplyResult):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
573
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
574 def __init__(self, cache, chunksize, length, callback):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
575 ApplyResult.__init__(self, cache, callback)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
576 self._success = True
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
577 self._value = [None] * length
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
578 self._chunksize = chunksize
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
579 if chunksize <= 0:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
580 self._number_left = 0
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
581 self._ready = True
77377
7ab7836894c4 Issue #12157: pool.map() does not handle empty iterable correctly
Richard Oudkerk <shibturn@gmail.com>
parents: 76712
diff changeset
582 del cache[self._job]
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
583 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
584 self._number_left = length//chunksize + bool(length % chunksize)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
585
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
586 def _set(self, i, success_result):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
587 success, result = success_result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
588 if success:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
589 self._value[i*self._chunksize:(i+1)*self._chunksize] = result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
590 self._number_left -= 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
591 if self._number_left == 0:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
592 if self._callback:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
593 self._callback(self._value)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
594 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
595 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
596 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
597 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
598 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
599 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
600 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
601
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
602 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
603 self._success = False
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
604 self._value = result
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
605 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
606 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
607 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
608 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
609 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
610 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
611 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
612
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
613 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
614 # Class whose instances are returned by `Pool.imap()`
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
615 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
616
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
617 class IMapIterator(object):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
618
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
619 def __init__(self, cache):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
620 self._cond = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
621 self._job = job_counter.next()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
622 self._cache = cache
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
623 self._items = collections.deque()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
624 self._index = 0
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
625 self._length = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
626 self._unsorted = {}
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
627 cache[self._job] = self
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
628
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
629 def __iter__(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
630 return self
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
631
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
632 def next(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
633 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
634 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
635 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
636 item = self._items.popleft()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
637 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
638 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
639 raise StopIteration
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
640 self._cond.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
641 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
642 item = self._items.popleft()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
643 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
644 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
645 raise StopIteration
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
646 raise TimeoutError
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
647 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
648 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
649
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
650 success, value = item
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
651 if success:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
652 return value
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
653 raise value
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
654
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
655 __next__ = next # XXX
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
656
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
657 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
658 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
659 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
660 if self._index == i:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
661 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
662 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
663 while self._index in self._unsorted:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
664 obj = self._unsorted.pop(self._index)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
665 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
666 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
667 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
668 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
669 self._unsorted[i] = obj
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
670
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
671 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
672 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
673 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
674 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
675
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
676 def _set_length(self, length):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
677 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
678 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
679 self._length = length
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
680 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
681 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
682 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
683 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
684 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
685
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
686 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
687 # Class whose instances are returned by `Pool.imap_unordered()`
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
688 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
689
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
690 class IMapUnorderedIterator(IMapIterator):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
691
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
692 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
693 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
694 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
695 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
696 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
697 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
698 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
699 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
700 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
701 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
702
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
703 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
704 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
705 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
706
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
707 class ThreadPool(Pool):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
708
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
709 from .dummy import Process
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
710
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
711 def __init__(self, processes=None, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
712 Pool.__init__(self, processes, initializer, initargs)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
713
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
714 def _setup_queues(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
715 self._inqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
716 self._outqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
717 self._quick_put = self._inqueue.put
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
718 self._quick_get = self._outqueue.get
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
719
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
720 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
721 def _help_stuff_finish(inqueue, task_handler, size):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
722 # put sentinels at head of inqueue to make workers finish
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
723 inqueue.not_empty.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
724 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
725 inqueue.queue.clear()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
726 inqueue.queue.extend([None] * size)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
727 inqueue.not_empty.notify_all()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
728 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
729 inqueue.not_empty.release()