cpython-lru_cache-weakref: Lib/multiprocessing/pool.py annotate

annotate Lib/multiprocessing/pool.py @ 58925:eecf1bef052d 2.6

Merged revisions 78207 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78207 | ezio.melotti | 2010-02-17 01:26:09 +0200 (Wed, 17 Feb 2010) | 1 line #7930: fix stripid ........

author	Ezio Melotti <ezio.melotti@gmail.com>
date	Tue, 16 Feb 2010 23:29:44 +0000
parents	5ef8b5a74efa
children	71f6129ddad4

rev	line source
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	1 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	2 # Module providing the `Pool` class for managing a process pool
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	3 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	4 # multiprocessing/pool.py
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	5 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	6 # Copyright (c) 2007-2008, R Oudkerk --- see COPYING.txt
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	7 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	8
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	9 __all__ = ['Pool']
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	10
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	11 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	12 # Imports
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	13 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	14
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	15 import threading
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	16 import Queue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	17 import itertools
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	18 import collections
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	19 import time
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	20
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	21 from multiprocessing import Process, cpu_count, TimeoutError
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	22 from multiprocessing.util import Finalize, debug
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	23
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	24 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	25 # Constants representing the state of a pool
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	26 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	27
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	28 RUN = 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	29 CLOSE = 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	30 TERMINATE = 2
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	31
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	32 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	33 # Miscellaneous
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	34 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	35
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	36 job_counter = itertools.count()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	37
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	38 def mapstar(args):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	39 return map(*args)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	40
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	41 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	42 # Code run by worker processes
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	43 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	44
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	45 def worker(inqueue, outqueue, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	46 put = outqueue.put
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	47 get = inqueue.get
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	48 if hasattr(inqueue, '_writer'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	49 inqueue._writer.close()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	50 outqueue._reader.close()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	51
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	52 if initializer is not None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	53 initializer(*initargs)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	54
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	55 while 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	56 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	57 task = get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	58 except (EOFError, IOError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	59 debug('worker got EOFError or IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	60 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	61
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	62 if task is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	63 debug('worker got sentinel -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	64 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	65
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	66 job, i, func, args, kwds = task
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	67 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	68 result = (True, func(args, *kwds))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	69 except Exception, e:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	70 result = (False, e)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	71 put((job, i, result))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	72
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	73 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	74 # Class representing a process pool
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	75 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	76
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	77 class Pool(object):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	78 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	79 Class which supports an async version of the `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	80 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	81 Process = Process
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	82
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	83 def __init__(self, processes=None, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	84 self._setup_queues()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	85 self._taskqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	86 self._cache = {}
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	87 self._state = RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	88
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	89 if processes is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	90 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	91 processes = cpu_count()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	92 except NotImplementedError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	93 processes = 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	94
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	95 self._pool = []
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	96 for i in range(processes):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	97 w = self.Process(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	98 target=worker,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	99 args=(self._inqueue, self._outqueue, initializer, initargs)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	100 )
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	101 self._pool.append(w)
48302 7ea805d879e1 issue3352: clean up the multiprocessing API to remove many get_/set_ methods and convert them to properties. Update the docs and the examples included. Jesse Noller <jnoller@gmail.com> parents: 48291 diff changeset	102 w.name = w.name.replace('Process', 'PoolWorker')
48291 28d93c9c3a03 patch up multiprocessing until it's API can be changed too Benjamin Peterson <benjamin@python.org> parents: 47696 diff changeset	103 w.daemon = True
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	104 w.start()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	105
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	106 self._task_handler = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	107 target=Pool._handle_tasks,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	108 args=(self._taskqueue, self._quick_put, self._outqueue, self._pool)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	109 )
48291 28d93c9c3a03 patch up multiprocessing until it's API can be changed too Benjamin Peterson <benjamin@python.org> parents: 47696 diff changeset	110 self._task_handler.daemon = True
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	111 self._task_handler._state = RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	112 self._task_handler.start()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	113
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	114 self._result_handler = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	115 target=Pool._handle_results,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	116 args=(self._outqueue, self._quick_get, self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	117 )
48291 28d93c9c3a03 patch up multiprocessing until it's API can be changed too Benjamin Peterson <benjamin@python.org> parents: 47696 diff changeset	118 self._result_handler.daemon = True
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	119 self._result_handler._state = RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	120 self._result_handler.start()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	121
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	122 self._terminate = Finalize(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	123 self, self._terminate_pool,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	124 args=(self._taskqueue, self._inqueue, self._outqueue, self._pool,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	125 self._task_handler, self._result_handler, self._cache),
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	126 exitpriority=15
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	127 )
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	128
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	129 def _setup_queues(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	130 from .queues import SimpleQueue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	131 self._inqueue = SimpleQueue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	132 self._outqueue = SimpleQueue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	133 self._quick_put = self._inqueue._writer.send
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	134 self._quick_get = self._outqueue._reader.recv
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	135
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	136 def apply(self, func, args=(), kwds={}):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	137 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	138 Equivalent of `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	139 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	140 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	141 return self.apply_async(func, args, kwds).get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	142
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	143 def map(self, func, iterable, chunksize=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	144 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	145 Equivalent of `map()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	146 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	147 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	148 return self.map_async(func, iterable, chunksize).get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	149
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	150 def imap(self, func, iterable, chunksize=1):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	151 '''
50002 5ef8b5a74efa Merged revisions 67245,67277,67289,67295,67301-67303,67307,67330,67332,67336,67355,67359,67362,67364,67367-67368,67370 via svnmerge from Georg Brandl <georg@python.org> parents: 48302 diff changeset	152 Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()`
47696 d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	153 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	154 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	155 if chunksize == 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	156 result = IMapIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	157 self._taskqueue.put((((result._job, i, func, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	158 for i, x in enumerate(iterable)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	159 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	160 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	161 assert chunksize > 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	162 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	163 result = IMapIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	164 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	165 for i, x in enumerate(task_batches)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	166 return (item for chunk in result for item in chunk)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	167
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	168 def imap_unordered(self, func, iterable, chunksize=1):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	169 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	170 Like `imap()` method but ordering of results is arbitrary
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	171 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	172 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	173 if chunksize == 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	174 result = IMapUnorderedIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	175 self._taskqueue.put((((result._job, i, func, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	176 for i, x in enumerate(iterable)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	177 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	178 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	179 assert chunksize > 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	180 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	181 result = IMapUnorderedIterator(self._cache)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	182 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	183 for i, x in enumerate(task_batches)), result._set_length))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	184 return (item for chunk in result for item in chunk)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	185
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	186 def apply_async(self, func, args=(), kwds={}, callback=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	187 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	188 Asynchronous equivalent of `apply()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	189 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	190 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	191 result = ApplyResult(self._cache, callback)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	192 self._taskqueue.put(([(result._job, None, func, args, kwds)], None))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	193 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	194
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	195 def map_async(self, func, iterable, chunksize=None, callback=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	196 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	197 Asynchronous equivalent of `map()` builtin
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	198 '''
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	199 assert self._state == RUN
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	200 if not hasattr(iterable, '__len__'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	201 iterable = list(iterable)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	202
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	203 if chunksize is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	204 chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	205 if extra:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	206 chunksize += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	207
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	208 task_batches = Pool._get_tasks(func, iterable, chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	209 result = MapResult(self._cache, chunksize, len(iterable), callback)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	210 self._taskqueue.put((((result._job, i, mapstar, (x,), {})
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	211 for i, x in enumerate(task_batches)), None))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	212 return result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	213
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	214 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	215 def _handle_tasks(taskqueue, put, outqueue, pool):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	216 thread = threading.current_thread()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	217
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	218 for taskseq, set_length in iter(taskqueue.get, None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	219 i = -1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	220 for i, task in enumerate(taskseq):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	221 if thread._state:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	222 debug('task handler found thread._state != RUN')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	223 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	224 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	225 put(task)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	226 except IOError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	227 debug('could not put task on queue')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	228 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	229 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	230 if set_length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	231 debug('doing set_length()')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	232 set_length(i+1)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	233 continue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	234 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	235 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	236 debug('task handler got sentinel')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	237
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	238
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	239 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	240 # tell result handler to finish when cache is empty
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	241 debug('task handler sending sentinel to result handler')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	242 outqueue.put(None)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	243
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	244 # tell workers there is no more work
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	245 debug('task handler sending sentinel to workers')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	246 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	247 put(None)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	248 except IOError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	249 debug('task handler got IOError when sending sentinels')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	250
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	251 debug('task handler exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	252
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	253 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	254 def _handle_results(outqueue, get, cache):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	255 thread = threading.current_thread()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	256
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	257 while 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	258 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	259 task = get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	260 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	261 debug('result handler got EOFError/IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	262 return
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	263
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	264 if thread._state:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	265 assert thread._state == TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	266 debug('result handler found thread._state=TERMINATE')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	267 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	268
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	269 if task is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	270 debug('result handler got sentinel')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	271 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	272
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	273 job, i, obj = task
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	274 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	275 cache[job]._set(i, obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	276 except KeyError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	277 pass
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	278
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	279 while cache and thread._state != TERMINATE:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	280 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	281 task = get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	282 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	283 debug('result handler got EOFError/IOError -- exiting')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	284 return
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	285
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	286 if task is None:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	287 debug('result handler ignoring extra sentinel')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	288 continue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	289 job, i, obj = task
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	290 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	291 cache[job]._set(i, obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	292 except KeyError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	293 pass
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	294
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	295 if hasattr(outqueue, '_reader'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	296 debug('ensuring that outqueue is not full')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	297 # If we don't make room available in outqueue then
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	298 # attempts to add the sentinel (None) to outqueue may
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	299 # block. There is guaranteed to be no more than 2 sentinels.
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	300 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	301 for i in range(10):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	302 if not outqueue._reader.poll():
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	303 break
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	304 get()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	305 except (IOError, EOFError):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	306 pass
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	307
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	308 debug('result handler exiting: len(cache)=%s, thread._state=%s',
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	309 len(cache), thread._state)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	310
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	311 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	312 def _get_tasks(func, it, size):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	313 it = iter(it)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	314 while 1:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	315 x = tuple(itertools.islice(it, size))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	316 if not x:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	317 return
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	318 yield (func, x)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	319
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	320 def __reduce__(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	321 raise NotImplementedError(
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	322 'pool objects cannot be passed between processes or pickled'
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	323 )
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	324
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	325 def close(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	326 debug('closing pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	327 if self._state == RUN:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	328 self._state = CLOSE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	329 self._taskqueue.put(None)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	330
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	331 def terminate(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	332 debug('terminating pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	333 self._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	334 self._terminate()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	335
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	336 def join(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	337 debug('joining pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	338 assert self._state in (CLOSE, TERMINATE)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	339 self._task_handler.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	340 self._result_handler.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	341 for p in self._pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	342 p.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	343
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	344 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	345 def _help_stuff_finish(inqueue, task_handler, size):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	346 # task_handler may be blocked trying to put items on inqueue
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	347 debug('removing tasks from inqueue until task handler finished')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	348 inqueue._rlock.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	349 while task_handler.is_alive() and inqueue._reader.poll():
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	350 inqueue._reader.recv()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	351 time.sleep(0)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	352
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	353 @classmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	354 def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	355 task_handler, result_handler, cache):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	356 # this is guaranteed to only be called once
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	357 debug('finalizing pool')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	358
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	359 task_handler._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	360 taskqueue.put(None) # sentinel
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	361
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	362 debug('helping task handler/workers to finish')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	363 cls._help_stuff_finish(inqueue, task_handler, len(pool))
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	364
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	365 assert result_handler.is_alive() or len(cache) == 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	366
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	367 result_handler._state = TERMINATE
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	368 outqueue.put(None) # sentinel
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	369
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	370 if pool and hasattr(pool[0], 'terminate'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	371 debug('terminating workers')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	372 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	373 p.terminate()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	374
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	375 debug('joining task handler')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	376 task_handler.join(1e100)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	377
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	378 debug('joining result handler')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	379 result_handler.join(1e100)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	380
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	381 if pool and hasattr(pool[0], 'terminate'):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	382 debug('joining pool workers')
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	383 for p in pool:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	384 p.join()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	385
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	386 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	387 # Class whose instances are returned by `Pool.apply_async()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	388 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	389
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	390 class ApplyResult(object):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	391
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	392 def __init__(self, cache, callback):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	393 self._cond = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	394 self._job = job_counter.next()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	395 self._cache = cache
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	396 self._ready = False
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	397 self._callback = callback
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	398 cache[self._job] = self
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	399
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	400 def ready(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	401 return self._ready
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	402
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	403 def successful(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	404 assert self._ready
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	405 return self._success
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	406
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	407 def wait(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	408 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	409 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	410 if not self._ready:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	411 self._cond.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	412 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	413 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	414
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	415 def get(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	416 self.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	417 if not self._ready:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	418 raise TimeoutError
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	419 if self._success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	420 return self._value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	421 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	422 raise self._value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	423
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	424 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	425 self._success, self._value = obj
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	426 if self._callback and self._success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	427 self._callback(self._value)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	428 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	429 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	430 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	431 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	432 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	433 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	434 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	435
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	436 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	437 # Class whose instances are returned by `Pool.map_async()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	438 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	439
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	440 class MapResult(ApplyResult):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	441
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	442 def __init__(self, cache, chunksize, length, callback):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	443 ApplyResult.__init__(self, cache, callback)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	444 self._success = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	445 self._value = [None] * length
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	446 self._chunksize = chunksize
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	447 if chunksize <= 0:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	448 self._number_left = 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	449 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	450 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	451 self._number_left = length//chunksize + bool(length % chunksize)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	452
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	453 def _set(self, i, success_result):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	454 success, result = success_result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	455 if success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	456 self._value[iself._chunksize:(i+1)self._chunksize] = result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	457 self._number_left -= 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	458 if self._number_left == 0:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	459 if self._callback:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	460 self._callback(self._value)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	461 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	462 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	463 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	464 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	465 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	466 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	467 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	468
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	469 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	470 self._success = False
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	471 self._value = result
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	472 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	473 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	474 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	475 self._ready = True
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	476 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	477 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	478 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	479
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	480 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	481 # Class whose instances are returned by `Pool.imap()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	482 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	483
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	484 class IMapIterator(object):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	485
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	486 def __init__(self, cache):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	487 self._cond = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	488 self._job = job_counter.next()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	489 self._cache = cache
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	490 self._items = collections.deque()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	491 self._index = 0
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	492 self._length = None
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	493 self._unsorted = {}
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	494 cache[self._job] = self
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	495
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	496 def __iter__(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	497 return self
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	498
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	499 def next(self, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	500 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	501 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	502 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	503 item = self._items.popleft()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	504 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	505 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	506 raise StopIteration
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	507 self._cond.wait(timeout)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	508 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	509 item = self._items.popleft()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	510 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	511 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	512 raise StopIteration
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	513 raise TimeoutError
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	514 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	515 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	516
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	517 success, value = item
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	518 if success:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	519 return value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	520 raise value
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	521
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	522 __next__ = next # XXX
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	523
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	524 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	525 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	526 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	527 if self._index == i:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	528 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	529 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	530 while self._index in self._unsorted:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	531 obj = self._unsorted.pop(self._index)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	532 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	533 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	534 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	535 else:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	536 self._unsorted[i] = obj
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	537
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	538 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	539 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	540 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	541 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	542
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	543 def _set_length(self, length):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	544 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	545 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	546 self._length = length
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	547 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	548 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	549 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	550 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	551 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	552
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	553 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	554 # Class whose instances are returned by `Pool.imap_unordered()`
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	555 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	556
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	557 class IMapUnorderedIterator(IMapIterator):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	558
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	559 def _set(self, i, obj):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	560 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	561 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	562 self._items.append(obj)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	563 self._index += 1
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	564 self._cond.notify()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	565 if self._index == self._length:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	566 del self._cache[self._job]
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	567 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	568 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	569
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	570 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	571 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	572 #
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	573
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	574 class ThreadPool(Pool):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	575
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	576 from .dummy import Process
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	577
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	578 def __init__(self, processes=None, initializer=None, initargs=()):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	579 Pool.__init__(self, processes, initializer, initargs)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	580
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	581 def _setup_queues(self):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	582 self._inqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	583 self._outqueue = Queue.Queue()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	584 self._quick_put = self._inqueue.put
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	585 self._quick_get = self._outqueue.get
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	586
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	587 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	588 def _help_stuff_finish(inqueue, task_handler, size):
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	589 # put sentinels at head of inqueue to make workers finish
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	590 inqueue.not_empty.acquire()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	591 try:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	592 inqueue.queue.clear()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	593 inqueue.queue.extend([None] * size)
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	594 inqueue.not_empty.notify_all()
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	595 finally:
d01165fae98e darn! I converted half of the files the wrong way. Benjamin Peterson <benjamin@python.org> parents: 47694 diff changeset	596 inqueue.not_empty.release()

Mercurial > cpython-lru_cache-weakref

annotate Lib/multiprocessing/pool.py @ 58925:eecf1bef052d 2.6