Mercurial > cpython-lru_cache-weakref

annotate Lib/multiprocessing/queues.py @ 79675:c0a2294de787 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16199: Backport 2to3 typo fix from 3.2.
author Chris Jerdonek <chris.jerdonek@gmail.com>
date Thu, 11 Oct 2012 16:15:46 -0700
parents e7b6dca28a2f
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
1 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
2 # Module implementing queues
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
3 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
4 # multiprocessing/queues.py
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
5 #
66887
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
6 # Copyright (c) 2006-2008, R Oudkerk
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
7 # All rights reserved.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
8 #
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
9 # Redistribution and use in source and binary forms, with or without
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
10 # modification, are permitted provided that the following conditions
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
11 # are met:
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
12 #
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
13 # 1. Redistributions of source code must retain the above copyright
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
14 # notice, this list of conditions and the following disclaimer.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
15 # 2. Redistributions in binary form must reproduce the above copyright
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
16 # notice, this list of conditions and the following disclaimer in the
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
17 # documentation and/or other materials provided with the distribution.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
18 # 3. Neither the name of author nor the names of any contributors may be
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
19 # used to endorse or promote products derived from this software
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
20 # without specific prior written permission.
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
21 #
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
22 # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
23 # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
24 # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
25 # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
26 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
27 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
28 # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
29 # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
30 # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
31 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
9ac8699d88d6 Merged revisions 87225 via svnmerge from
R. David Murray <rdmurray@bitdance.com>
parents: 57385
diff changeset
32 # SUCH DAMAGE.
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
33 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
34
52506
26b1ad09d1a7 merge 70792 to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 48289
diff changeset
35 __all__ = ['Queue', 'SimpleQueue', 'JoinableQueue']
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
36
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
37 import sys
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
38 import os
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
39 import threading
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
40 import collections
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
41 import time
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
42 import atexit
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
43 import weakref
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
44
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
45 from Queue import Empty, Full
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
46 import _multiprocessing
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
47 from multiprocessing import Pipe
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
48 from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
49 from multiprocessing.util import debug, info, Finalize, register_after_fork
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
50 from multiprocessing.forking import assert_spawning
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
51
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
52 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
53 # Queue type using a pipe, buffer and thread
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
54 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
55
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
56 class Queue(object):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
57
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
58 def __init__(self, maxsize=0):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
59 if maxsize <= 0:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
60 maxsize = _multiprocessing.SemLock.SEM_VALUE_MAX
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
61 self._maxsize = maxsize
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
62 self._reader, self._writer = Pipe(duplex=False)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
63 self._rlock = Lock()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
64 self._opid = os.getpid()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
65 if sys.platform == 'win32':
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
66 self._wlock = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
67 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
68 self._wlock = Lock()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
69 self._sem = BoundedSemaphore(maxsize)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
70
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
71 self._after_fork()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
72
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
73 if sys.platform != 'win32':
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
74 register_after_fork(self, Queue._after_fork)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
75
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
76 def __getstate__(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
77 assert_spawning(self)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
78 return (self._maxsize, self._reader, self._writer,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
79 self._rlock, self._wlock, self._sem, self._opid)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
80
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
81 def __setstate__(self, state):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
82 (self._maxsize, self._reader, self._writer,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
83 self._rlock, self._wlock, self._sem, self._opid) = state
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
84 self._after_fork()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
85
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
86 def _after_fork(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
87 debug('Queue._after_fork()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
88 self._notempty = threading.Condition(threading.Lock())
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
89 self._buffer = collections.deque()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
90 self._thread = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
91 self._jointhread = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
92 self._joincancelled = False
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
93 self._closed = False
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
94 self._close = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
95 self._send = self._writer.send
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
96 self._recv = self._reader.recv
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
97 self._poll = self._reader.poll
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
98
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
99 def put(self, obj, block=True, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
100 assert not self._closed
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
101 if not self._sem.acquire(block, timeout):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
102 raise Full
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
103
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
104 self._notempty.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
105 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
106 if self._thread is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
107 self._start_thread()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
108 self._buffer.append(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
109 self._notempty.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
110 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
111 self._notempty.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
112
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
113 def get(self, block=True, timeout=None):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
114 if block and timeout is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
115 self._rlock.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
116 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
117 res = self._recv()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
118 self._sem.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
119 return res
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
120 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
121 self._rlock.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
122
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
123 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
124 if block:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
125 deadline = time.time() + timeout
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
126 if not self._rlock.acquire(block, timeout):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
127 raise Empty
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
128 try:
73471
e7b6dca28a2f Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
Antoine Pitrou <solipsis@pitrou.net>
parents: 72063
diff changeset
129 if block:
e7b6dca28a2f Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
Antoine Pitrou <solipsis@pitrou.net>
parents: 72063
diff changeset
130 timeout = deadline - time.time()
e7b6dca28a2f Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
Antoine Pitrou <solipsis@pitrou.net>
parents: 72063
diff changeset
131 if timeout < 0 or not self._poll(timeout):
e7b6dca28a2f Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
Antoine Pitrou <solipsis@pitrou.net>
parents: 72063
diff changeset
132 raise Empty
e7b6dca28a2f Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
Antoine Pitrou <solipsis@pitrou.net>
parents: 72063
diff changeset
133 elif not self._poll():
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
134 raise Empty
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
135 res = self._recv()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
136 self._sem.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
137 return res
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
138 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
139 self._rlock.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
140
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
141 def qsize(self):
56830
97d7cd365dd5 Merged revisions 73206,73232,73299,73683,74020,74185,74544,74643,74647,74817,74838-74839,74865,74946,75402,75459,75604,75696 via svnmerge from
Georg Brandl <georg@python.org>
parents: 55555
diff changeset
142 # Raises NotImplementedError on Mac OSX because of broken sem_getvalue()
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
143 return self._maxsize - self._sem._semlock._get_value()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
144
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
145 def empty(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
146 return not self._poll()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
147
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
148 def full(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
149 return self._sem._semlock._is_zero()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
150
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
151 def get_nowait(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
152 return self.get(False)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
153
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
154 def put_nowait(self, obj):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
155 return self.put(obj, False)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
156
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
157 def close(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
158 self._closed = True
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
159 self._reader.close()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
160 if self._close:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
161 self._close()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
162
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
163 def join_thread(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
164 debug('Queue.join_thread()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
165 assert self._closed
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
166 if self._jointhread:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
167 self._jointhread()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
168
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
169 def cancel_join_thread(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
170 debug('Queue.cancel_join_thread()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
171 self._joincancelled = True
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
172 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
173 self._jointhread.cancel()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
174 except AttributeError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
175 pass
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
176
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
177 def _start_thread(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
178 debug('Queue._start_thread()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
179
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
180 # Start thread which transfers data from buffer to pipe
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
181 self._buffer.clear()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
182 self._thread = threading.Thread(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
183 target=Queue._feed,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
184 args=(self._buffer, self._notempty, self._send,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
185 self._wlock, self._writer.close),
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
186 name='QueueFeederThread'
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
187 )
48289
024b2b984866 change a few uses of the threading APIs
Benjamin Peterson <benjamin@python.org>
parents: 47696
diff changeset
188 self._thread.daemon = True
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
189
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
190 debug('doing self._thread.start()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
191 self._thread.start()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
192 debug('... done self._thread.start()')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
193
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
194 # On process exit we will wait for data to be flushed to pipe.
72063
d316315a8781 Issue #4106: Fix occasional exceptions printed out by multiprocessing on interpreter shutdown.
Antoine Pitrou <solipsis@pitrou.net>
parents: 68256
diff changeset
195 if not self._joincancelled:
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
196 self._jointhread = Finalize(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
197 self._thread, Queue._finalize_join,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
198 [weakref.ref(self._thread)],
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
199 exitpriority=-5
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
200 )
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
201
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
202 # Send sentinel to the thread queue object when garbage collected
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
203 self._close = Finalize(
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
204 self, Queue._finalize_close,
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
205 [self._buffer, self._notempty],
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
206 exitpriority=10
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
207 )
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
208
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
209 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
210 def _finalize_join(twr):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
211 debug('joining queue thread')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
212 thread = twr()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
213 if thread is not None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
214 thread.join()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
215 debug('... queue thread joined')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
216 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
217 debug('... queue thread already dead')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
218
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
219 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
220 def _finalize_close(buffer, notempty):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
221 debug('telling queue thread to quit')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
222 notempty.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
223 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
224 buffer.append(_sentinel)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
225 notempty.notify()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
226 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
227 notempty.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
228
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
229 @staticmethod
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
230 def _feed(buffer, notempty, send, writelock, close):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
231 debug('starting thread to feed data to pipe')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
232 from .util import is_exiting
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
233
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
234 nacquire = notempty.acquire
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
235 nrelease = notempty.release
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
236 nwait = notempty.wait
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
237 bpopleft = buffer.popleft
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
238 sentinel = _sentinel
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
239 if sys.platform != 'win32':
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
240 wacquire = writelock.acquire
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
241 wrelease = writelock.release
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
242 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
243 wacquire = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
244
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
245 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
246 while 1:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
247 nacquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
248 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
249 if not buffer:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
250 nwait()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
251 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
252 nrelease()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
253 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
254 while 1:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
255 obj = bpopleft()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
256 if obj is sentinel:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
257 debug('feeder thread got sentinel -- exiting')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
258 close()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
259 return
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
260
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
261 if wacquire is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
262 send(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
263 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
264 wacquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
265 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
266 send(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
267 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
268 wrelease()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
269 except IndexError:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
270 pass
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
271 except Exception, e:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
272 # Since this runs in a daemon thread the resources it uses
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
273 # may be become unusable while the process is cleaning up.
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
274 # We ignore errors which happen after the process has
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
275 # started to cleanup.
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
276 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
277 if is_exiting():
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
278 info('error in queue thread: %s', e)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
279 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
280 import traceback
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
281 traceback.print_exc()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
282 except Exception:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
283 pass
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
284
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
285 _sentinel = object()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
286
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
287 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
288 # A queue type which also supports join() and task_done() methods
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
289 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
290 # Note that if you do not call task_done() for each finished task then
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
291 # eventually the counter's semaphore may overflow causing Bad Things
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
292 # to happen.
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
293 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
294
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
295 class JoinableQueue(Queue):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
296
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
297 def __init__(self, maxsize=0):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
298 Queue.__init__(self, maxsize)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
299 self._unfinished_tasks = Semaphore(0)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
300 self._cond = Condition()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
301
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
302 def __getstate__(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
303 return Queue.__getstate__(self) + (self._cond, self._unfinished_tasks)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
304
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
305 def __setstate__(self, state):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
306 Queue.__setstate__(self, state[:-2])
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
307 self._cond, self._unfinished_tasks = state[-2:]
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
308
55555
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
309 def put(self, obj, block=True, timeout=None):
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
310 assert not self._closed
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
311 if not self._sem.acquire(block, timeout):
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
312 raise Full
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
313
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
314 self._notempty.acquire()
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
315 self._cond.acquire()
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
316 try:
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
317 if self._thread is None:
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
318 self._start_thread()
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
319 self._buffer.append(obj)
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
320 self._unfinished_tasks.release()
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
321 self._notempty.notify()
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
322 finally:
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
323 self._cond.release()
65d1682dcd5e Merge fix for 4660 back to 26 maint
Jesse Noller <jnoller@gmail.com>
parents: 52506
diff changeset
324 self._notempty.release()
47696
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
325
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
326 def task_done(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
327 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
328 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
329 if not self._unfinished_tasks.acquire(False):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
330 raise ValueError('task_done() called too many times')
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
331 if self._unfinished_tasks._semlock._is_zero():
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
332 self._cond.notify_all()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
333 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
334 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
335
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
336 def join(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
337 self._cond.acquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
338 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
339 if not self._unfinished_tasks._semlock._is_zero():
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
340 self._cond.wait()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
341 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
342 self._cond.release()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
343
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
344 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
345 # Simplified Queue type -- really just a locked pipe
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
346 #
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
347
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
348 class SimpleQueue(object):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
349
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
350 def __init__(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
351 self._reader, self._writer = Pipe(duplex=False)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
352 self._rlock = Lock()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
353 if sys.platform == 'win32':
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
354 self._wlock = None
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
355 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
356 self._wlock = Lock()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
357 self._make_methods()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
358
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
359 def empty(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
360 return not self._reader.poll()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
361
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
362 def __getstate__(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
363 assert_spawning(self)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
364 return (self._reader, self._writer, self._rlock, self._wlock)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
365
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
366 def __setstate__(self, state):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
367 (self._reader, self._writer, self._rlock, self._wlock) = state
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
368 self._make_methods()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
369
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
370 def _make_methods(self):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
371 recv = self._reader.recv
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
372 racquire, rrelease = self._rlock.acquire, self._rlock.release
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
373 def get():
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
374 racquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
375 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
376 return recv()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
377 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
378 rrelease()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
379 self.get = get
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
380
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
381 if self._wlock is None:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
382 # writes to a message oriented win32 pipe are atomic
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
383 self.put = self._writer.send
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
384 else:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
385 send = self._writer.send
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
386 wacquire, wrelease = self._wlock.acquire, self._wlock.release
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
387 def put(obj):
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
388 wacquire()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
389 try:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
390 return send(obj)
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
391 finally:
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
392 wrelease()
d01165fae98e darn! I converted half of the files the wrong way.
Benjamin Peterson <benjamin@python.org>
parents: 47694
diff changeset
393 self.put = put