cpython-lru_cache-weakref: Lib/Queue.py annotate

annotate Lib/Queue.py @ 80661:94de77bd0b4b 2.7

Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.

author	Andrew Svetlov <andrew.svetlov@gmail.com>
date	Thu, 29 Nov 2012 19:51:28 +0200
parents	b36cb4602e21
children	737b53ec5d1a

rev	line source
11691 e34991022571 Mass patch by Ka-Ping Yee: Guido van Rossum <guido@python.org> parents: 11267 diff changeset	1 """A multi-producer, multi-consumer queue."""
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	2
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	3 from time import time as _time
52184 e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	4 try:
e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	5 import threading as _threading
e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	6 except ImportError:
e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	7 import dummy_threading as _threading
31436 63fa60f2d61c * Move collections.deque() in from the sandbox Raymond Hettinger <python@rcn.com> parents: 29711 diff changeset	8 from collections import deque
44714 aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	9 import heapq
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	10
44714 aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	11 __all__ = ['Empty', 'Full', 'Queue', 'PriorityQueue', 'LifoQueue']
29711 58132c813d7b Add __all__ . Brett Cannon <bcannon@gmail.com> parents: 26892 diff changeset	12
15902 eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	13 class Empty(Exception):
eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	14 "Exception raised by Queue.get(block=0)/get_nowait()."
eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	15 pass
eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	16
eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	17 class Full(Exception):
eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	18 "Exception raised by Queue.put(block=0)/put_nowait()."
eef321b6e41c Variant of Skip's patch 103246 (Remove unneeded string exception compat from Queue). Tim Peters <tim.peters@gmail.com> parents: 13050 diff changeset	19 pass
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	20
f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	21 class Queue:
38717 b23eeb802780 document the class, not its initializer Skip Montanaro <skip@pobox.com> parents: 37207 diff changeset	22 """Create a queue object with a given maximum size.
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	23
38717 b23eeb802780 document the class, not its initializer Skip Montanaro <skip@pobox.com> parents: 37207 diff changeset	24 If maxsize is <= 0, the queue size is infinite.
b23eeb802780 document the class, not its initializer Skip Montanaro <skip@pobox.com> parents: 37207 diff changeset	25 """
b23eeb802780 document the class, not its initializer Skip Montanaro <skip@pobox.com> parents: 37207 diff changeset	26 def __init__(self, maxsize=0):
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	27 self.maxsize = maxsize
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	28 self._init(maxsize)
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	29 # mutex must be held whenever the queue is mutating. All methods
73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	30 # that acquire mutex must release it before returning. mutex
40392 193146e4d276 ... and the number of the counting shall be three. Raymond Hettinger <python@rcn.com> parents: 38717 diff changeset	31 # is shared between the three conditions, so acquiring and
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	32 # releasing the conditions also acquires and releases mutex.
52184 e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	33 self.mutex = _threading.Lock()
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	34 # Notify not_empty whenever an item is added to the queue; a
73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	35 # thread waiting to get is notified then.
52184 e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	36 self.not_empty = _threading.Condition(self.mutex)
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	37 # Notify not_full whenever an item is removed from the queue;
73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	38 # a thread waiting to put is notified then.
52184 e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	39 self.not_full = _threading.Condition(self.mutex)
37198 bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	40 # Notify all_tasks_done whenever the number of unfinished tasks
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	41 # drops to zero; thread waiting to join() is notified to resume
52184 e4d9844aba25 Move the previously local import of threading to module level. Georg Brandl <georg@python.org> parents: 47598 diff changeset	42 self.all_tasks_done = _threading.Condition(self.mutex)
37198 bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	43 self.unfinished_tasks = 0
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	44
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	45 def task_done(self):
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	46 """Indicate that a formerly enqueued task is complete.
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	47
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	48 Used by Queue consumer threads. For each get() used to fetch a task,
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	49 a subsequent call to task_done() tells the queue that the processing
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	50 on the task is complete.
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	51
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	52 If a join() is currently blocking, it will resume when all items
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	53 have been processed (meaning that a task_done() call was received
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	54 for every item that had been put() into the queue).
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	55
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	56 Raises a ValueError if called more times than there were items
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	57 placed in the queue.
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	58 """
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	59 self.all_tasks_done.acquire()
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	60 try:
37207 96e4d53128c9 Don't decrement below zero. And add more tests. Raymond Hettinger <python@rcn.com> parents: 37205 diff changeset	61 unfinished = self.unfinished_tasks - 1
37198 bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	62 if unfinished <= 0:
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	63 if unfinished < 0:
37205 49622d65c29d Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 37198 diff changeset	64 raise ValueError('task_done() called too many times')
47598 c349f9fbc704 give the threading API PEP 8 names Benjamin Peterson <benjamin@python.org> parents: 47248 diff changeset	65 self.all_tasks_done.notify_all()
37207 96e4d53128c9 Don't decrement below zero. And add more tests. Raymond Hettinger <python@rcn.com> parents: 37205 diff changeset	66 self.unfinished_tasks = unfinished
37198 bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	67 finally:
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	68 self.all_tasks_done.release()
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	69
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	70 def join(self):
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	71 """Blocks until all items in the Queue have been gotten and processed.
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	72
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	73 The count of unfinished tasks goes up whenever an item is added to the
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	74 queue. The count goes down whenever a consumer thread calls task_done()
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	75 to indicate the item was retrieved and all work on it is complete.
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	76
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	77 When the count of unfinished tasks drops to zero, join() unblocks.
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	78 """
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	79 self.all_tasks_done.acquire()
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	80 try:
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	81 while self.unfinished_tasks:
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	82 self.all_tasks_done.wait()
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	83 finally:
bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	84 self.all_tasks_done.release()
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	85
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	86 def qsize(self):
10120 5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	87 """Return the approximate size of the queue (not reliable!)."""
8197 3a55b4c5e3c7 Replace all calls to acquire_lock() and release_lock() with acquire() Guido van Rossum <guido@python.org> parents: 7926 diff changeset	88 self.mutex.acquire()
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	89 n = self._qsize()
8197 3a55b4c5e3c7 Replace all calls to acquire_lock() and release_lock() with acquire() Guido van Rossum <guido@python.org> parents: 7926 diff changeset	90 self.mutex.release()
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	91 return n
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	92
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	93 def empty(self):
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	94 """Return True if the queue is empty, False otherwise (not reliable!)."""
8197 3a55b4c5e3c7 Replace all calls to acquire_lock() and release_lock() with acquire() Guido van Rossum <guido@python.org> parents: 7926 diff changeset	95 self.mutex.acquire()
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	96 n = not self._qsize()
8197 3a55b4c5e3c7 Replace all calls to acquire_lock() and release_lock() with acquire() Guido van Rossum <guido@python.org> parents: 7926 diff changeset	97 self.mutex.release()
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	98 return n
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	99
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	100 def full(self):
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	101 """Return True if the queue is full, False otherwise (not reliable!)."""
8197 3a55b4c5e3c7 Replace all calls to acquire_lock() and release_lock() with acquire() Guido van Rossum <guido@python.org> parents: 7926 diff changeset	102 self.mutex.acquire()
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	103 n = 0 < self.maxsize == self._qsize()
8197 3a55b4c5e3c7 Replace all calls to acquire_lock() and release_lock() with acquire() Guido van Rossum <guido@python.org> parents: 7926 diff changeset	104 self.mutex.release()
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	105 return n
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	106
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	107 def put(self, item, block=True, timeout=None):
7926 e29d435ab142 Clarify that put blocks when the queue is full. Add some blank Guido van Rossum <guido@python.org> parents: 7755 diff changeset	108 """Put an item into the queue.
e29d435ab142 Clarify that put blocks when the queue is full. Add some blank Guido van Rossum <guido@python.org> parents: 7755 diff changeset	109
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	110 If optional args 'block' is true and 'timeout' is None (the default),
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	111 block if necessary until a free slot is available. If 'timeout' is
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	112 a positive number, it blocks at most 'timeout' seconds and raises
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	113 the Full exception if no free slot was available within that time.
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	114 Otherwise ('block' is false), put an item on the queue if a free slot
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	115 is immediately available, else raise the Full exception ('timeout'
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	116 is ignored in that case).
10120 5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	117 """
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	118 self.not_full.acquire()
73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	119 try:
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	120 if self.maxsize > 0:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	121 if not block:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	122 if self._qsize() == self.maxsize:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	123 raise Full
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	124 elif timeout is None:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	125 while self._qsize() == self.maxsize:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	126 self.not_full.wait()
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	127 elif timeout < 0:
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	128 raise ValueError("'timeout' must be a positive number")
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	129 else:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	130 endtime = _time() + timeout
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	131 while self._qsize() == self.maxsize:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	132 remaining = endtime - _time()
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	133 if remaining <= 0.0:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	134 raise Full
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	135 self.not_full.wait(remaining)
23160 c548e726b3bc Fix bug 544473 - "Queue module can deadlock". Mark Hammond <mhammond@skippinet.com.au> parents: 16657 diff changeset	136 self._put(item)
37198 bed3f52f0dc6 SF Patch #1455676: Simplify using Queues with daemon consumer threads Raymond Hettinger <python@rcn.com> parents: 32625 diff changeset	137 self.unfinished_tasks += 1
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	138 self.not_empty.notify()
23160 c548e726b3bc Fix bug 544473 - "Queue module can deadlock". Mark Hammond <mhammond@skippinet.com.au> parents: 16657 diff changeset	139 finally:
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	140 self.not_full.release()
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	141
10120 5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	142 def put_nowait(self, item):
5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	143 """Put an item into the queue without blocking.
5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	144
5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	145 Only enqueue the item if a free slot is immediately available.
5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	146 Otherwise raise the Full exception.
5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	147 """
32625 d42efc2b45e9 Simplified the new get/get_nowait/put/put_nowait implementations a bit. Tim Peters <tim.peters@gmail.com> parents: 32624 diff changeset	148 return self.put(item, False)
7926 e29d435ab142 Clarify that put blocks when the queue is full. Add some blank Guido van Rossum <guido@python.org> parents: 7755 diff changeset	149
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	150 def get(self, block=True, timeout=None):
10120 5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	151 """Remove and return an item from the queue.
5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	152
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	153 If optional args 'block' is true and 'timeout' is None (the default),
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	154 block if necessary until an item is available. If 'timeout' is
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	155 a positive number, it blocks at most 'timeout' seconds and raises
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	156 the Empty exception if no item was available within that time.
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	157 Otherwise ('block' is false), return an item if one is immediately
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	158 available, else raise the Empty exception ('timeout' is ignored
3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	159 in that case).
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	160 """
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	161 self.not_empty.acquire()
73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	162 try:
32625 d42efc2b45e9 Simplified the new get/get_nowait/put/put_nowait implementations a bit. Tim Peters <tim.peters@gmail.com> parents: 32624 diff changeset	163 if not block:
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	164 if not self._qsize():
32625 d42efc2b45e9 Simplified the new get/get_nowait/put/put_nowait implementations a bit. Tim Peters <tim.peters@gmail.com> parents: 32624 diff changeset	165 raise Empty
d42efc2b45e9 Simplified the new get/get_nowait/put/put_nowait implementations a bit. Tim Peters <tim.peters@gmail.com> parents: 32624 diff changeset	166 elif timeout is None:
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	167 while not self._qsize():
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	168 self.not_empty.wait()
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	169 elif timeout < 0:
c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	170 raise ValueError("'timeout' must be a positive number")
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	171 else:
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	172 endtime = _time() + timeout
44696 c790587f2992 Refactor if/elif chain for clarity and speed. Remove dependency on subclasses having to implement _empty and _full. Raymond Hettinger <python@rcn.com> parents: 40392 diff changeset	173 while not self._qsize():
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	174 remaining = endtime - _time()
32625 d42efc2b45e9 Simplified the new get/get_nowait/put/put_nowait implementations a bit. Tim Peters <tim.peters@gmail.com> parents: 32624 diff changeset	175 if remaining <= 0.0:
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	176 raise Empty
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	177 self.not_empty.wait(remaining)
23160 c548e726b3bc Fix bug 544473 - "Queue module can deadlock". Mark Hammond <mhammond@skippinet.com.au> parents: 16657 diff changeset	178 item = self._get()
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	179 self.not_full.notify()
73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	180 return item
23160 c548e726b3bc Fix bug 544473 - "Queue module can deadlock". Mark Hammond <mhammond@skippinet.com.au> parents: 16657 diff changeset	181 finally:
32624 73016bd01a71 Bug #788520: Queue class has logic error when non-blocking Tim Peters <tim.peters@gmail.com> parents: 31436 diff changeset	182 self.not_empty.release()
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	183
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	184 def get_nowait(self):
10120 5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	185 """Remove and return an item from the queue without blocking.
7926 e29d435ab142 Clarify that put blocks when the queue is full. Add some blank Guido van Rossum <guido@python.org> parents: 7755 diff changeset	186
25796 3eafbde56b21 Patch #572628: Optional timeouts for put and get. Martin v. Löwis <martin@v.loewis.de> parents: 24058 diff changeset	187 Only get an item if one is immediately available. Otherwise
10120 5b15642e7bdf Tim Peters: Guido van Rossum <guido@python.org> parents: 8197 diff changeset	188 raise the Empty exception.
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	189 """
32625 d42efc2b45e9 Simplified the new get/get_nowait/put/put_nowait implementations a bit. Tim Peters <tim.peters@gmail.com> parents: 32624 diff changeset	190 return self.get(False)
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	191
4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	192 # Override these methods to implement other queue organizations
4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	193 # (e.g. stack or priority queue).
4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	194 # These will only be called with appropriate locks held
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	195
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	196 # Initialize the queue representation
4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	197 def _init(self, maxsize):
31436 63fa60f2d61c * Move collections.deque() in from the sandbox Raymond Hettinger <python@rcn.com> parents: 29711 diff changeset	198 self.queue = deque()
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	199
44714 aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	200 def _qsize(self, len=len):
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	201 return len(self.queue)
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	202
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	203 # Put a new item in the queue
4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	204 def _put(self, item):
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6612 diff changeset	205 self.queue.append(item)
995 f025eef93130 New module implementing a multi-everything queue. Guido van Rossum <guido@python.org> parents: diff changeset	206
6612 4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	207 # Get an item from the queue
4b2d8dc92db4 (Queue.Empty): When class based exceptions are in force, derive this Barry Warsaw <barry@python.org> parents: 1463 diff changeset	208 def _get(self):
31436 63fa60f2d61c * Move collections.deque() in from the sandbox Raymond Hettinger <python@rcn.com> parents: 29711 diff changeset	209 return self.queue.popleft()
44714 aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	210
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	211
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	212 class PriorityQueue(Queue):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	213 '''Variant of Queue that retrieves open entries in priority order (lowest first).
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	214
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	215 Entries are typically tuples of the form: (priority number, data).
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	216 '''
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	217
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	218 def _init(self, maxsize):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	219 self.queue = []
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	220
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	221 def _qsize(self, len=len):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	222 return len(self.queue)
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	223
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	224 def _put(self, item, heappush=heapq.heappush):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	225 heappush(self.queue, item)
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	226
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	227 def _get(self, heappop=heapq.heappop):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	228 return heappop(self.queue)
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	229
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	230
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	231 class LifoQueue(Queue):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	232 '''Variant of Queue that retrieves most recently added entries first.'''
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	233
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	234 def _init(self, maxsize):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	235 self.queue = []
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	236
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	237 def _qsize(self, len=len):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	238 return len(self.queue)
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	239
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	240 def _put(self, item):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	241 self.queue.append(item)
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	242
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	243 def _get(self):
aa5cf6394641 Add queues will alternative fetch orders (priority based and stack based). Raymond Hettinger <python@rcn.com> parents: 44696 diff changeset	244 return self.queue.pop()

Mercurial > cpython-lru_cache-weakref

annotate Lib/Queue.py @ 80661:94de77bd0b4b 2.7