Mercurial > cpython-lru_cache-weakref

annotate Lib/queue.py @ 84007:dbfbdf2b5c19 3.3

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix #17967: For ftp urls CWD to target instead of hopping to each directory towards target. This fixes a bug where target is accessible, but parent directories are restricted.
author Senthil Kumaran <senthil@uthcode.com>
date Sat, 01 Jun 2013 08:27:06 -0700
parents 76d2e0761d18
children 2122d56d6bc5 8d28d44f3a9a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
1 '''A multi-producer, multi-consumer queue.'''
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
2
52259
e41f5a89f5c0 port the queue change r70405
Benjamin Peterson <benjamin@python.org>
parents: 52045
diff changeset
3 try:
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
4 import threading
52259
e41f5a89f5c0 port the queue change r70405
Benjamin Peterson <benjamin@python.org>
parents: 52045
diff changeset
5 except ImportError:
74309
32eae3c48631 Fix nasty typo
Raymond Hettinger <python@rcn.com>
parents: 74307
diff changeset
6 import dummy_threading as threading
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
7 from collections import deque
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
8 from heapq import heappush, heappop
76597
76d2e0761d18 Issue #14428, #14397: Implement the PEP 418
Victor Stinner <victor.stinner@gmail.com>
parents: 75675
diff changeset
9 try:
76d2e0761d18 Issue #14428, #14397: Implement the PEP 418
Victor Stinner <victor.stinner@gmail.com>
parents: 75675
diff changeset
10 from time import monotonic as time
76d2e0761d18 Issue #14428, #14397: Implement the PEP 418
Victor Stinner <victor.stinner@gmail.com>
parents: 75675
diff changeset
11 except ImportError:
76d2e0761d18 Issue #14428, #14397: Implement the PEP 418
Victor Stinner <victor.stinner@gmail.com>
parents: 75675
diff changeset
12 from time import time
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
13
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
14 __all__ = ['Empty', 'Full', 'Queue', 'PriorityQueue', 'LifoQueue']
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
15
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
16 class Empty(Exception):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
17 'Exception raised by Queue.get(block=0)/get_nowait().'
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
18 pass
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
19
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
20 class Full(Exception):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
21 'Exception raised by Queue.put(block=0)/put_nowait().'
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
22 pass
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
23
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
24 class Queue:
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
25 '''Create a queue object with a given maximum size.
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
26
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
27 If maxsize is <= 0, the queue size is infinite.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
28 '''
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
29
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
30 def __init__(self, maxsize=0):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
31 self.maxsize = maxsize
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
32 self._init(maxsize)
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
33
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
34 # mutex must be held whenever the queue is mutating. All methods
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
35 # that acquire mutex must release it before returning. mutex
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
36 # is shared between the three conditions, so acquiring and
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
37 # releasing the conditions also acquires and releases mutex.
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
38 self.mutex = threading.Lock()
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
39
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
40 # Notify not_empty whenever an item is added to the queue; a
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
41 # thread waiting to get is notified then.
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
42 self.not_empty = threading.Condition(self.mutex)
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
43
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
44 # Notify not_full whenever an item is removed from the queue;
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
45 # a thread waiting to put is notified then.
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
46 self.not_full = threading.Condition(self.mutex)
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
47
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
48 # Notify all_tasks_done whenever the number of unfinished tasks
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
49 # drops to zero; thread waiting to join() is notified to resume
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
50 self.all_tasks_done = threading.Condition(self.mutex)
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
51 self.unfinished_tasks = 0
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
52
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
53 def task_done(self):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
54 '''Indicate that a formerly enqueued task is complete.
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
55
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
56 Used by Queue consumer threads. For each get() used to fetch a task,
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
57 a subsequent call to task_done() tells the queue that the processing
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
58 on the task is complete.
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
59
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
60 If a join() is currently blocking, it will resume when all items
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
61 have been processed (meaning that a task_done() call was received
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
62 for every item that had been put() into the queue).
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
63
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
64 Raises a ValueError if called more times than there were items
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
65 placed in the queue.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
66 '''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
67 with self.all_tasks_done:
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
68 unfinished = self.unfinished_tasks - 1
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
69 if unfinished <= 0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
70 if unfinished < 0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
71 raise ValueError('task_done() called too many times')
47615
045f83a1b864 Merged revisions 64125 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 46899
diff changeset
72 self.all_tasks_done.notify_all()
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
73 self.unfinished_tasks = unfinished
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
74
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
75 def join(self):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
76 '''Blocks until all items in the Queue have been gotten and processed.
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
77
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
78 The count of unfinished tasks goes up whenever an item is added to the
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
79 queue. The count goes down whenever a consumer thread calls task_done()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
80 to indicate the item was retrieved and all work on it is complete.
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
81
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
82 When the count of unfinished tasks drops to zero, join() unblocks.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
83 '''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
84 with self.all_tasks_done:
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
85 while self.unfinished_tasks:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
86 self.all_tasks_done.wait()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
87
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
88 def qsize(self):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
89 '''Return the approximate size of the queue (not reliable!).'''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
90 with self.mutex:
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
91 return self._qsize()
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
92
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
93 def empty(self):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
94 '''Return True if the queue is empty, False otherwise (not reliable!).
52045
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
95
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
96 This method is likely to be removed at some point. Use qsize() == 0
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
97 as a direct substitute, but be aware that either approach risks a race
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
98 condition where a queue can grow before the result of empty() or
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
99 qsize() can be used.
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
100
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
101 To create code that needs to wait for all queued tasks to be
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
102 completed, the preferred technique is to use the join() method.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
103 '''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
104 with self.mutex:
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
105 return not self._qsize()
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
106
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
107 def full(self):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
108 '''Return True if the queue is full, False otherwise (not reliable!).
52045
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
109
65831
bc6cce4ab3c6 Issue 10110: Let Queue.put recognize a full queue when the maxsize parameter has been reduced.
Raymond Hettinger <python@rcn.com>
parents: 52259
diff changeset
110 This method is likely to be removed at some point. Use qsize() >= n
52045
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
111 as a direct substitute, but be aware that either approach risks a race
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
112 condition where a queue can shrink before the result of full() or
aef5cfe73358 Document the suggested alternative to emtpy() and full().
Raymond Hettinger <python@rcn.com>
parents: 47615
diff changeset
113 qsize() can be used.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
114 '''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
115 with self.mutex:
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
116 return 0 < self.maxsize <= self._qsize()
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
117
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
118 def put(self, item, block=True, timeout=None):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
119 '''Put an item into the queue.
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
120
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
121 If optional args 'block' is true and 'timeout' is None (the default),
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
122 block if necessary until a free slot is available. If 'timeout' is
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
123 a positive number, it blocks at most 'timeout' seconds and raises
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
124 the Full exception if no free slot was available within that time.
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
125 Otherwise ('block' is false), put an item on the queue if a free slot
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
126 is immediately available, else raise the Full exception ('timeout'
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
127 is ignored in that case).
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
128 '''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
129 with self.not_full:
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
130 if self.maxsize > 0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
131 if not block:
65831
bc6cce4ab3c6 Issue 10110: Let Queue.put recognize a full queue when the maxsize parameter has been reduced.
Raymond Hettinger <python@rcn.com>
parents: 52259
diff changeset
132 if self._qsize() >= self.maxsize:
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
133 raise Full
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
134 elif timeout is None:
65831
bc6cce4ab3c6 Issue 10110: Let Queue.put recognize a full queue when the maxsize parameter has been reduced.
Raymond Hettinger <python@rcn.com>
parents: 52259
diff changeset
135 while self._qsize() >= self.maxsize:
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
136 self.not_full.wait()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
137 elif timeout < 0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
138 raise ValueError("'timeout' must be a positive number")
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
139 else:
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
140 endtime = time() + timeout
65831
bc6cce4ab3c6 Issue 10110: Let Queue.put recognize a full queue when the maxsize parameter has been reduced.
Raymond Hettinger <python@rcn.com>
parents: 52259
diff changeset
141 while self._qsize() >= self.maxsize:
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
142 remaining = endtime - time()
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
143 if remaining <= 0.0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
144 raise Full
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
145 self.not_full.wait(remaining)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
146 self._put(item)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
147 self.unfinished_tasks += 1
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
148 self.not_empty.notify()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
149
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
150 def get(self, block=True, timeout=None):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
151 '''Remove and return an item from the queue.
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
152
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
153 If optional args 'block' is true and 'timeout' is None (the default),
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
154 block if necessary until an item is available. If 'timeout' is
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
155 a positive number, it blocks at most 'timeout' seconds and raises
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
156 the Empty exception if no item was available within that time.
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
157 Otherwise ('block' is false), return an item if one is immediately
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
158 available, else raise the Empty exception ('timeout' is ignored
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
159 in that case).
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
160 '''
74298
c68426b241ca Simplify the code using with-statements.
Raymond Hettinger <python@rcn.com>
parents: 65831
diff changeset
161 with self.not_empty:
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
162 if not block:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
163 if not self._qsize():
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
164 raise Empty
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
165 elif timeout is None:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
166 while not self._qsize():
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
167 self.not_empty.wait()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
168 elif timeout < 0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
169 raise ValueError("'timeout' must be a positive number")
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
170 else:
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
171 endtime = time() + timeout
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
172 while not self._qsize():
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
173 remaining = endtime - time()
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
174 if remaining <= 0.0:
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
175 raise Empty
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
176 self.not_empty.wait(remaining)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
177 item = self._get()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
178 self.not_full.notify()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
179 return item
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
180
74306
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
181 def put_nowait(self, item):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
182 '''Put an item into the queue without blocking.
74306
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
183
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
184 Only enqueue the item if a free slot is immediately available.
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
185 Otherwise raise the Full exception.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
186 '''
74306
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
187 return self.put(item, block=False)
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
188
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
189 def get_nowait(self):
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
190 '''Remove and return an item from the queue without blocking.
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
191
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
192 Only get an item if one is immediately available. Otherwise
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
193 raise the Empty exception.
74307
ca2a35140e6a Make the docstring style consistent.
Raymond Hettinger <python@rcn.com>
parents: 74306
diff changeset
194 '''
74306
07608e67bc01 Improve clarity with keyword argument for block. Move nowait methods together.
Raymond Hettinger <python@rcn.com>
parents: 74305
diff changeset
195 return self.get(block=False)
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
196
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
197 # Override these methods to implement other queue organizations
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
198 # (e.g. stack or priority queue).
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
199 # These will only be called with appropriate locks held
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
200
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
201 # Initialize the queue representation
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
202 def _init(self, maxsize):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
203 self.queue = deque()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
204
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
205 def _qsize(self):
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
206 return len(self.queue)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
207
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
208 # Put a new item in the queue
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
209 def _put(self, item):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
210 self.queue.append(item)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
211
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
212 # Get an item from the queue
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
213 def _get(self):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
214 return self.queue.popleft()
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
215
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
216
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
217 class PriorityQueue(Queue):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
218 '''Variant of Queue that retrieves open entries in priority order (lowest first).
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
219
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
220 Entries are typically tuples of the form: (priority number, data).
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
221 '''
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
222
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
223 def _init(self, maxsize):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
224 self.queue = []
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
225
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
226 def _qsize(self):
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
227 return len(self.queue)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
228
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
229 def _put(self, item):
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
230 heappush(self.queue, item)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
231
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
232 def _get(self):
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
233 return heappop(self.queue)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
234
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
235
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
236 class LifoQueue(Queue):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
237 '''Variant of Queue that retrieves most recently added entries first.'''
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
238
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
239 def _init(self, maxsize):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
240 self.queue = []
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
241
74305
190ad17f5a87 Minor code clean-ups and beautifications.
Raymond Hettinger <python@rcn.com>
parents: 74298
diff changeset
242 def _qsize(self):
46899
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
243 return len(self.queue)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
244
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
245 def _put(self, item):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
246 self.queue.append(item)
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
247
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
248 def _get(self):
bd2c95c165ea Rename Queue module to queue.
Alexandre Vassalotti <alexandre@peadrop.com>
parents:
diff changeset
249 return self.queue.pop()