cpython-lru_cache-weakref: Lib/multiprocessing/pool.py annotate

annotate Lib/multiprocessing/pool.py @ 54294:dba767bc3155 legacy-trunk

remove mention of old ctypes version

author	Benjamin Peterson <benjamin@python.org>
date	Sat, 23 May 2009 20:59:09 +0000
parents	71f6129ddad4
children	3370d608e5aa

rev	line source
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	1 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	2 # Module providing the `Pool` class for managing a process pool
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	3 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	4 # multiprocessing/pool.py
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	5 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	6 # Copyright (c) 2007-2008, R Oudkerk --- see COPYING.txt
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	7 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	8
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	9 __all__ = ['Pool']
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	10
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	11 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	12 # Imports
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	13 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	14
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	15 import threading
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	16 import Queue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	17 import itertools
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	18 import collections
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	19 import time
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	20
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	21 from multiprocessing import Process, cpu_count, TimeoutError
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	22 from multiprocessing.util import Finalize, debug
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	23
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	24 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	25 # Constants representing the state of a pool
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	26 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	27
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	28 RUN = 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	29 CLOSE = 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	30 TERMINATE = 2
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	31
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	32 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	33 # Miscellaneous
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	34 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	35
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	36 job_counter = itertools.count()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	37
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	38 def mapstar(args):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	39 return map(*args)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	40
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	41 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	42 # Code run by worker processes
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	43 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	44
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	45 def worker(inqueue, outqueue, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	46 put = outqueue.put
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	47 get = inqueue.get
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	48 if hasattr(inqueue, '_writer'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	49 inqueue._writer.close()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	50 outqueue._reader.close()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	51
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	52 if initializer is not None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	53 initializer(*initargs)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	54
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	55 while 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	56 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	57 task = get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	58 except (EOFError, IOError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	59 debug('worker got EOFError or IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	60 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	61
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	62 if task is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	63 debug('worker got sentinel -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	64 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	65
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	66 job, i, func, args, kwds = task
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	67 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	68 result = (True, func(args, *kwds))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	69 except Exception, e:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	70 result = (False, e)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	71 put((job, i, result))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	72
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	73 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	74 # Class representing a process pool
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	75 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	76
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	77 class Pool(object):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	78 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	79 Class which supports an async version of the `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	80 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	81 Process = Process
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	82
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	83 def __init__(self, processes=None, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	84 self._setup_queues()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	85 self._taskqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	86 self._cache = {}
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	87 self._state = RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	88
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	89 if processes is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	90 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	91 processes = cpu_count()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	92 except NotImplementedError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	93 processes = 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	94
52727 71f6129ddad4 Add custom initializer argument to multiprocess.Manager, courtesy of lekma Jesse Noller <jnoller@gmail.com>* parents: 50002 diff changeset	95 if initializer is not None and not hasattr(initializer, '__call__'):
71f6129ddad4 Add custom initializer argument to multiprocess.Manager, courtesy of lekma Jesse Noller <jnoller@gmail.com>* parents: 50002 diff changeset	96 raise TypeError('initializer must be a callable')
71f6129ddad4 Add custom initializer argument to multiprocess.Manager, courtesy of lekma Jesse Noller <jnoller@gmail.com>* parents: 50002 diff changeset	97
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	98 self._pool = []
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	99 for i in range(processes):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	100 w = self.Process(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	101 target=worker,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	102 args=(self._inqueue, self._outqueue, initializer, initargs)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	103 )
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	104 self._pool.append(w)
48302 7ea805d879e1 issue3352: clean up the multiprocessing API to remove many get_/set_ methods and convert them to properties. Update the docs and the examples included. Jesse Noller <jnoller@gmail.com> parents: 48291 diff changeset	105 w.name = w.name.replace('Process', 'PoolWorker')
48291 28d93c9c3a03 patch up multiprocessing until it's API can be changed too Benjamin Peterson <benjamin@python.org> parents: 47696 diff changeset	106 w.daemon = True
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	107 w.start()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	108
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	109 self._task_handler = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	110 target=Pool._handle_tasks,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	111 args=(self._taskqueue, self._quick_put, self._outqueue, self._pool)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	112 )
48291 28d93c9c3a03 patch up multiprocessing until it's API can be changed too Benjamin Peterson <benjamin@python.org> parents: 47696 diff changeset	113 self._task_handler.daemon = True
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	114 self._task_handler._state = RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	115 self._task_handler.start()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	116
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	117 self._result_handler = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	118 target=Pool._handle_results,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	119 args=(self._outqueue, self._quick_get, self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	120 )
48291 28d93c9c3a03 patch up multiprocessing until it's API can be changed too Benjamin Peterson <benjamin@python.org> parents: 47696 diff changeset	121 self._result_handler.daemon = True
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	122 self._result_handler._state = RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	123 self._result_handler.start()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	124
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	125 self._terminate = Finalize(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	126 self, self._terminate_pool,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	127 args=(self._taskqueue, self._inqueue, self._outqueue, self._pool,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	128 self._task_handler, self._result_handler, self._cache),
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	129 exitpriority=15
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	130 )
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	131
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	132 def _setup_queues(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	133 from .queues import SimpleQueue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	134 self._inqueue = SimpleQueue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	135 self._outqueue = SimpleQueue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	136 self._quick_put = self._inqueue._writer.send
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	137 self._quick_get = self._outqueue._reader.recv
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	138
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	139 def apply(self, func, args=(), kwds={}):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	140 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	141 Equivalent of `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	142 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	143 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	144 return self.apply_async(func, args, kwds).get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	145
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	146 def map(self, func, iterable, chunksize=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	147 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	148 Equivalent of `map()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	149 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	150 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	151 return self.map_async(func, iterable, chunksize).get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	152
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	153 def imap(self, func, iterable, chunksize=1):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	154 '''
50002 5ef8b5a74efa Merged revisions 67245,67277,67289,67295,67301-67303,67307,67330,67332,67336,67355,67359,67362,67364,67367-67368,67370 via svnmerge from Georg Brandl <georg@python.org> parents: 48302 diff changeset	155 Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()`
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	156 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	157 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	158 if chunksize == 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	159 result = IMapIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	160 self._taskqueue.put((((result._job, i, func, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	161 for i, x in enumerate(iterable)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	162 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	163 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	164 assert chunksize > 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	165 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	166 result = IMapIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	167 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	168 for i, x in enumerate(task_batches)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	169 return (item for chunk in result for item in chunk)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	170
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	171 def imap_unordered(self, func, iterable, chunksize=1):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	172 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	173 Like `imap()` method but ordering of results is arbitrary
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	174 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	175 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	176 if chunksize == 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	177 result = IMapUnorderedIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	178 self._taskqueue.put((((result._job, i, func, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	179 for i, x in enumerate(iterable)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	180 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	181 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	182 assert chunksize > 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	183 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	184 result = IMapUnorderedIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	185 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	186 for i, x in enumerate(task_batches)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	187 return (item for chunk in result for item in chunk)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	188
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	189 def apply_async(self, func, args=(), kwds={}, callback=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	190 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	191 Asynchronous equivalent of `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	192 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	193 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	194 result = ApplyResult(self._cache, callback)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	195 self._taskqueue.put(([(result._job, None, func, args, kwds)], None))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	196 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	197
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	198 def map_async(self, func, iterable, chunksize=None, callback=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	199 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	200 Asynchronous equivalent of `map()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	201 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	202 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	203 if not hasattr(iterable, '__len__'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	204 iterable = list(iterable)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	205
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	206 if chunksize is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	207 chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	208 if extra:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	209 chunksize += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	210
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	211 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	212 result = MapResult(self._cache, chunksize, len(iterable), callback)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	213 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	214 for i, x in enumerate(task_batches)), None))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	215 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	216
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	217 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	218 def _handle_tasks(taskqueue, put, outqueue, pool):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	219 thread = threading.current_thread()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	220
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	221 for taskseq, set_length in iter(taskqueue.get, None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	222 i = -1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	223 for i, task in enumerate(taskseq):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	224 if thread._state:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	225 debug('task handler found thread._state != RUN')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	226 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	227 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	228 put(task)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	229 except IOError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	230 debug('could not put task on queue')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	231 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	232 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	233 if set_length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	234 debug('doing set_length()')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	235 set_length(i+1)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	236 continue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	237 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	238 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	239 debug('task handler got sentinel')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	240
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	241
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	242 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	243 # tell result handler to finish when cache is empty
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	244 debug('task handler sending sentinel to result handler')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	245 outqueue.put(None)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	246
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	247 # tell workers there is no more work
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	248 debug('task handler sending sentinel to workers')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	249 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	250 put(None)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	251 except IOError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	252 debug('task handler got IOError when sending sentinels')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	253
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	254 debug('task handler exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	255
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	256 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	257 def _handle_results(outqueue, get, cache):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	258 thread = threading.current_thread()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	259
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	260 while 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	261 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	262 task = get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	263 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	264 debug('result handler got EOFError/IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	265 return
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	266
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	267 if thread._state:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	268 assert thread._state == TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	269 debug('result handler found thread._state=TERMINATE')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	270 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	271
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	272 if task is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	273 debug('result handler got sentinel')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	274 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	275
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	276 job, i, obj = task
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	277 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	278 cache[job]._set(i, obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	279 except KeyError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	280 pass
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	281
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	282 while cache and thread._state != TERMINATE:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	283 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	284 task = get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	285 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	286 debug('result handler got EOFError/IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	287 return
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	288
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	289 if task is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	290 debug('result handler ignoring extra sentinel')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	291 continue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	292 job, i, obj = task
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	293 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	294 cache[job]._set(i, obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	295 except KeyError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	296 pass
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	297
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	298 if hasattr(outqueue, '_reader'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	299 debug('ensuring that outqueue is not full')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	300 # If we don't make room available in outqueue then
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	301 # attempts to add the sentinel (None) to outqueue may
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	302 # block. There is guaranteed to be no more than 2 sentinels.
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	303 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	304 for i in range(10):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	305 if not outqueue._reader.poll():
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	306 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	307 get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	308 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	309 pass
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	310
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	311 debug('result handler exiting: len(cache)=%s, thread._state=%s',
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	312 len(cache), thread._state)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	313
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	314 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	315 def _get_tasks(func, it, size):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	316 it = iter(it)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	317 while 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	318 x = tuple(itertools.islice(it, size))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	319 if not x:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	320 return
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	321 yield (func, x)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	322
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	323 def __reduce__(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	324 raise NotImplementedError(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	325 'pool objects cannot be passed between processes or pickled'
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	326 )
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	327
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	328 def close(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	329 debug('closing pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	330 if self._state == RUN:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	331 self._state = CLOSE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	332 self._taskqueue.put(None)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	333
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	334 def terminate(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	335 debug('terminating pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	336 self._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	337 self._terminate()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	338
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	339 def join(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	340 debug('joining pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	341 assert self._state in (CLOSE, TERMINATE)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	342 self._task_handler.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	343 self._result_handler.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	344 for p in self._pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	345 p.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	346
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	347 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	348 def _help_stuff_finish(inqueue, task_handler, size):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	349 # task_handler may be blocked trying to put items on inqueue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	350 debug('removing tasks from inqueue until task handler finished')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	351 inqueue._rlock.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	352 while task_handler.is_alive() and inqueue._reader.poll():
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	353 inqueue._reader.recv()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	354 time.sleep(0)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	355
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	356 @classmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	357 def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	358 task_handler, result_handler, cache):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	359 # this is guaranteed to only be called once
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	360 debug('finalizing pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	361
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	362 task_handler._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	363 taskqueue.put(None) # sentinel
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	364
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	365 debug('helping task handler/workers to finish')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	366 cls._help_stuff_finish(inqueue, task_handler, len(pool))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	367
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	368 assert result_handler.is_alive() or len(cache) == 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	369
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	370 result_handler._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	371 outqueue.put(None) # sentinel
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	372
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	373 if pool and hasattr(pool[0], 'terminate'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	374 debug('terminating workers')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	375 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	376 p.terminate()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	377
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	378 debug('joining task handler')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	379 task_handler.join(1e100)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	380
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	381 debug('joining result handler')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	382 result_handler.join(1e100)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	383
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	384 if pool and hasattr(pool[0], 'terminate'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	385 debug('joining pool workers')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	386 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	387 p.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	388
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	389 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	390 # Class whose instances are returned by `Pool.apply_async()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	391 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	392
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	393 class ApplyResult(object):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	394
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	395 def __init__(self, cache, callback):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	396 self._cond = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	397 self._job = job_counter.next()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	398 self._cache = cache
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	399 self._ready = False
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	400 self._callback = callback
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	401 cache[self._job] = self
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	402
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	403 def ready(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	404 return self._ready
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	405
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	406 def successful(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	407 assert self._ready
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	408 return self._success
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	409
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	410 def wait(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	411 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	412 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	413 if not self._ready:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	414 self._cond.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	415 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	416 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	417
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	418 def get(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	419 self.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	420 if not self._ready:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	421 raise TimeoutError
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	422 if self._success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	423 return self._value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	424 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	425 raise self._value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	426
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	427 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	428 self._success, self._value = obj
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	429 if self._callback and self._success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	430 self._callback(self._value)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	431 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	432 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	433 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	434 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	435 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	436 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	437 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	438
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	439 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	440 # Class whose instances are returned by `Pool.map_async()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	441 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	442
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	443 class MapResult(ApplyResult):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	444
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	445 def __init__(self, cache, chunksize, length, callback):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	446 ApplyResult.__init__(self, cache, callback)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	447 self._success = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	448 self._value = [None] * length
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	449 self._chunksize = chunksize
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	450 if chunksize <= 0:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	451 self._number_left = 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	452 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	453 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	454 self._number_left = length//chunksize + bool(length % chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	455
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	456 def _set(self, i, success_result):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	457 success, result = success_result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	458 if success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	459 self._value[iself._chunksize:(i+1)self._chunksize] = result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	460 self._number_left -= 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	461 if self._number_left == 0:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	462 if self._callback:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	463 self._callback(self._value)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	464 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	465 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	466 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	467 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	468 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	469 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	470 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	471
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	472 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	473 self._success = False
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	474 self._value = result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	475 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	476 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	477 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	478 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	479 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	480 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	481 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	482
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	483 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	484 # Class whose instances are returned by `Pool.imap()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	485 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	486
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	487 class IMapIterator(object):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	488
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	489 def __init__(self, cache):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	490 self._cond = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	491 self._job = job_counter.next()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	492 self._cache = cache
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	493 self._items = collections.deque()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	494 self._index = 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	495 self._length = None
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	496 self._unsorted = {}
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	497 cache[self._job] = self
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	498
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	499 def __iter__(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	500 return self
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	501
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	502 def next(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	503 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	504 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	505 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	506 item = self._items.popleft()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	507 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	508 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	509 raise StopIteration
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	510 self._cond.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	511 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	512 item = self._items.popleft()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	513 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	514 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	515 raise StopIteration
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	516 raise TimeoutError
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	517 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	518 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	519
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	520 success, value = item
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	521 if success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	522 return value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	523 raise value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	524
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	525 __next__ = next # XXX
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	526
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	527 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	528 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	529 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	530 if self._index == i:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	531 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	532 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	533 while self._index in self._unsorted:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	534 obj = self._unsorted.pop(self._index)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	535 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	536 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	537 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	538 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	539 self._unsorted[i] = obj
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	540
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	541 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	542 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	543 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	544 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	545
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	546 def _set_length(self, length):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	547 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	548 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	549 self._length = length
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	550 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	551 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	552 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	553 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	554 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	555
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	556 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	557 # Class whose instances are returned by `Pool.imap_unordered()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	558 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	559
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	560 class IMapUnorderedIterator(IMapIterator):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	561
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	562 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	563 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	564 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	565 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	566 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	567 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	568 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	569 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	570 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	571 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	572
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	573 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	574 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	575 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	576
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	577 class ThreadPool(Pool):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	578
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	579 from .dummy import Process
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	580
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	581 def __init__(self, processes=None, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	582 Pool.__init__(self, processes, initializer, initargs)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	583
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	584 def _setup_queues(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	585 self._inqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	586 self._outqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	587 self._quick_put = self._inqueue.put
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	588 self._quick_get = self._outqueue.get
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	589
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	590 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	591 def _help_stuff_finish(inqueue, task_handler, size):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	592 # put sentinels at head of inqueue to make workers finish
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	593 inqueue.not_empty.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	594 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	595 inqueue.queue.clear()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	596 inqueue.queue.extend([None] * size)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	597 inqueue.not_empty.notify_all()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	598 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	599 inqueue.not_empty.release()

Mercurial > cpython-lru_cache-weakref

annotate Lib/multiprocessing/pool.py @ 54294:dba767bc3155 legacy-trunk