Mercurial > cpython-lru_cache-weakref
annotate Lib/urllib2.py @ 54294:dba767bc3155 legacy-trunk
remove mention of old ctypes version
| author | Benjamin Peterson <benjamin@python.org> |
|---|---|
| date | Sat, 23 May 2009 20:59:09 +0000 |
| parents | d69769b4fd20 |
| children | e7b4a443f764 |
| rev | line source |
|---|---|
|
11707
6f4146846337
The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents:
11669
diff
changeset
|
1 """An extensible library for opening URLs using a variety of protocols |
| 11669 | 2 |
| 3 The simplest way to use this module is to call the urlopen function, | |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
4 which accepts a string containing a URL or a Request object (described |
| 11669 | 5 below). It opens the URL and returns the results as file-like |
| 6 object; the returned object has some extra methods described below. | |
| 7 | |
| 25731 | 8 The OpenerDirector manages a collection of Handler objects that do |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
9 all the actual work. Each Handler implements a particular protocol or |
| 11669 | 10 option. The OpenerDirector is a composite object that invokes the |
| 11 Handlers needed to open the requested URL. For example, the | |
| 12 HTTPHandler performs HTTP GET and POST requests and deals with | |
| 13 non-error returns. The HTTPRedirectHandler automatically deals with | |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
14 HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
15 deals with digest authentication. |
| 11669 | 16 |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
17 urlopen(url, data=None) -- Basic usage is the same as original |
| 11669 | 18 urllib. pass the url and optionally data to post to an HTTP URL, and |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
19 get a file-like object back. One difference is that you can also pass |
| 11669 | 20 a Request instance instead of URL. Raises a URLError (subclass of |
| 21 IOError); for HTTP errors, raises an HTTPError, which can also be | |
| 22 treated as a valid response. | |
| 23 | |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
24 build_opener -- Function that creates a new OpenerDirector instance. |
|
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
25 Will install the default handlers. Accepts one or more Handlers as |
| 11669 | 26 arguments, either instances or Handler classes that it will |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
27 instantiate. If one of the argument is a subclass of the default |
| 11669 | 28 handler, the argument will be installed instead of the default. |
| 29 | |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
30 install_opener -- Installs a new opener as the default opener. |
| 11669 | 31 |
| 32 objects of interest: | |
|
41133
01355124ce40
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
41048
diff
changeset
|
33 OpenerDirector -- |
| 11669 | 34 |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
35 Request -- An object that encapsulates the state of a request. The |
|
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
36 state can be as simple as the URL. It can also include extra HTTP |
| 11669 | 37 headers, e.g. a User-Agent. |
| 38 | |
| 39 BaseHandler -- | |
| 40 | |
| 41 exceptions: | |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
42 URLError -- A subclass of IOError, individual protocols have their own |
|
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
43 specific subclass. |
| 11669 | 44 |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
45 HTTPError -- Also a valid HTTP response, so you can treat an HTTP error |
|
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
46 as an exceptional event or valid response. |
| 11669 | 47 |
| 48 internals: | |
| 49 BaseHandler and parent | |
| 50 _call_chain conventions | |
| 51 | |
| 52 Example usage: | |
| 53 | |
| 54 import urllib2 | |
| 55 | |
| 56 # set up authentication info | |
| 57 authinfo = urllib2.HTTPBasicAuthHandler() | |
|
41514
91f312176eaf
Fix SF #1703110, Incorrect example for add_password() (use uri, not host)
Neal Norwitz <nnorwitz@gmail.com>
parents:
41510
diff
changeset
|
58 authinfo.add_password(realm='PDQ Application', |
|
91f312176eaf
Fix SF #1703110, Incorrect example for add_password() (use uri, not host)
Neal Norwitz <nnorwitz@gmail.com>
parents:
41510
diff
changeset
|
59 uri='https://mahler:8092/site-updates.py', |
|
91f312176eaf
Fix SF #1703110, Incorrect example for add_password() (use uri, not host)
Neal Norwitz <nnorwitz@gmail.com>
parents:
41510
diff
changeset
|
60 user='klem', |
|
91f312176eaf
Fix SF #1703110, Incorrect example for add_password() (use uri, not host)
Neal Norwitz <nnorwitz@gmail.com>
parents:
41510
diff
changeset
|
61 passwd='geheim$parole') |
| 11669 | 62 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
63 proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"}) |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
64 |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
65 # build a new opener that adds authentication and caching FTP handlers |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
66 opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler) |
| 11669 | 67 |
| 68 # install it | |
| 69 urllib2.install_opener(opener) | |
| 70 | |
| 71 f = urllib2.urlopen('http://www.python.org/') | |
| 72 | |
| 73 | |
| 74 """ | |
| 75 | |
| 76 # XXX issues: | |
| 77 # If an authentication error handler that tries to perform | |
|
20916
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
78 # authentication for some reason but fails, how should the error be |
|
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
79 # signalled? The client needs to know the HTTP error code. But if |
|
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
80 # the handler knows that the problem was, e.g., that it didn't know |
|
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
81 # that hash algo that requested in the challenge, it would be good to |
|
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
82 # pass that information along to the client, too. |
|
37401
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
83 # ftp errors aren't handled cleanly |
|
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
84 # check digest against correct (i.e. non-apache) implementation |
| 11669 | 85 |
|
37401
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
86 # Possible extensions: |
|
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
87 # complex proxies XXX not sure what exactly was meant by this |
| 11669 | 88 # abstract factory for opener |
| 89 | |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
90 import base64 |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
91 import hashlib |
| 11669 | 92 import httplib |
| 93 import mimetools | |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
94 import os |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
95 import posixpath |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
96 import random |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
97 import re |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
98 import socket |
| 11669 | 99 import sys |
| 100 import time | |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
101 import urlparse |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
102 import bisect |
| 11669 | 103 |
| 104 try: | |
| 105 from cStringIO import StringIO | |
| 106 except ImportError: | |
| 107 from StringIO import StringIO | |
| 108 | |
|
37403
260ee391f908
Readd urllib.quote import as it doesn't cause any harm.
Georg Brandl <georg@python.org>
parents:
37401
diff
changeset
|
109 from urllib import (unwrap, unquote, splittype, splithost, quote, |
|
48278
62c69d88e985
Update __all__ for cookielib, csv, os, and urllib2 for objects imported into
Brett Cannon <bcannon@gmail.com>
parents:
48259
diff
changeset
|
110 addinfourl, splitport, |
|
62c69d88e985
Update __all__ for cookielib, csv, os, and urllib2 for objects imported into
Brett Cannon <bcannon@gmail.com>
parents:
48259
diff
changeset
|
111 splitattr, ftpwrapper, splituser, splitpasswd, splitvalue) |
| 11669 | 112 |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
113 # support for FileHandler, proxies via environment variables |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
114 from urllib import localhost, url2pathname, getproxies |
| 11669 | 115 |
|
37400
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
116 # used in User-Agent header sent |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
117 __version__ = sys.version[:3] |
| 11669 | 118 |
| 119 _opener = None | |
|
47372
e2c5ae30cffe
Fixed the semantic of timeout for socket.create_connection and
Facundo Batista <facundobatista@gmail.com>
parents:
46662
diff
changeset
|
120 def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT): |
| 11669 | 121 global _opener |
| 122 if _opener is None: | |
| 123 _opener = build_opener() | |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
124 return _opener.open(url, data, timeout) |
| 11669 | 125 |
| 126 def install_opener(opener): | |
| 127 global _opener | |
| 128 _opener = opener | |
| 129 | |
| 130 # do these error classes make sense? | |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
131 # make sure all of the IOError stuff is overridden. we just want to be |
|
24783
01d6c5146ff9
Remove ugly irregular spaces from in front of some comments.
Fred Drake <fdrake@acm.org>
parents:
24263
diff
changeset
|
132 # subtypes. |
| 11669 | 133 |
| 134 class URLError(IOError): | |
| 135 # URLError is a sub-type of IOError, but it doesn't share any of | |
|
30654
055a80881e96
SF patch [ 816787 ] urllib2.URLError don't calll IOError.__init__
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29823
diff
changeset
|
136 # the implementation. need to override __init__ and __str__. |
|
055a80881e96
SF patch [ 816787 ] urllib2.URLError don't calll IOError.__init__
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29823
diff
changeset
|
137 # It sets self.args for compatibility with other EnvironmentError |
|
055a80881e96
SF patch [ 816787 ] urllib2.URLError don't calll IOError.__init__
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29823
diff
changeset
|
138 # subclasses, but args doesn't have the typical format with errno in |
|
055a80881e96
SF patch [ 816787 ] urllib2.URLError don't calll IOError.__init__
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29823
diff
changeset
|
139 # slot 0 and strerror in slot 1. This may be better than nothing. |
| 11669 | 140 def __init__(self, reason): |
|
30654
055a80881e96
SF patch [ 816787 ] urllib2.URLError don't calll IOError.__init__
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29823
diff
changeset
|
141 self.args = reason, |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
142 self.reason = reason |
| 11669 | 143 |
| 144 def __str__(self): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
145 return '<urlopen error %s>' % self.reason |
| 11669 | 146 |
| 147 class HTTPError(URLError, addinfourl): | |
| 148 """Raised when HTTP error occurs, but also acts like non-error return""" | |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
149 __super_init = addinfourl.__init__ |
| 11669 | 150 |
| 151 def __init__(self, url, code, msg, hdrs, fp): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
152 self.code = code |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
153 self.msg = msg |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
154 self.hdrs = hdrs |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
155 self.fp = fp |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
156 self.filename = url |
|
23646
220ed7febb00
Fix HTTPError __init__ for cases where fp is None.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
23645
diff
changeset
|
157 # The addinfourl classes depend on fp being a valid file |
|
220ed7febb00
Fix HTTPError __init__ for cases where fp is None.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
23645
diff
changeset
|
158 # object. In some cases, the HTTPError may not have a valid |
|
220ed7febb00
Fix HTTPError __init__ for cases where fp is None.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
23645
diff
changeset
|
159 # file object. If this happens, the simplest workaround is to |
|
24263
e6033254df21
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
24125
diff
changeset
|
160 # not initialize the base classes. |
|
23646
220ed7febb00
Fix HTTPError __init__ for cases where fp is None.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
23645
diff
changeset
|
161 if fp is not None: |
|
46371
b3cd0339ca77
#2585: initialize code attribute of HTTPError.
Georg Brandl <georg@python.org>
parents:
45980
diff
changeset
|
162 self.__super_init(fp, hdrs, url, code) |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
163 |
| 11669 | 164 def __str__(self): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
165 return 'HTTP Error %s: %s' % (self.code, self.msg) |
| 11669 | 166 |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
167 # copied from cookielib.py |
| 38211 | 168 _cut_port_re = re.compile(r":\d+$") |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
169 def request_host(request): |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
170 """Return request-host, as defined by RFC 2965. |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
171 |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
172 Variation from RFC: returned value is lowercased, for convenient |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
173 comparison. |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
174 |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
175 """ |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
176 url = request.get_full_url() |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
177 host = urlparse.urlparse(url)[1] |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
178 if host == "": |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
179 host = request.get_header("Host", "") |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
180 |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
181 # remove port, if present |
| 38211 | 182 host = _cut_port_re.sub("", host, 1) |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
183 return host.lower() |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
184 |
| 11669 | 185 class Request: |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
186 |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
187 def __init__(self, url, data=None, headers={}, |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
188 origin_req_host=None, unverifiable=False): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
189 # unwrap('<URL:type://host/path>') --> 'type://host/path' |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
190 self.__original = unwrap(url) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
191 self.type = None |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
192 # self.__r_type is what's left after doing the splittype |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
193 self.host = None |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
194 self.port = None |
| 11669 | 195 self.data = data |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
196 self.headers = {} |
|
29154
5e6d0dcf155f
Revert some changes back to dict.items made in a previous patch.
Brett Cannon <bcannon@gmail.com>
parents:
29080
diff
changeset
|
197 for key, value in headers.items(): |
|
29079
f57751ffd6e7
Change Request.add_header to call string.capitalize in order to normalize
Brett Cannon <bcannon@gmail.com>
parents:
28973
diff
changeset
|
198 self.add_header(key, value) |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
199 self.unredirected_hdrs = {} |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
200 if origin_req_host is None: |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
201 origin_req_host = request_host(self) |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
202 self.origin_req_host = origin_req_host |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
203 self.unverifiable = unverifiable |
| 11669 | 204 |
| 205 def __getattr__(self, attr): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
206 # XXX this is a fallback mechanism to guard against these |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
207 # methods getting called in a non-standard order. this may be |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
208 # too complicated and/or unnecessary. |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
209 # XXX should the __r_XXX attributes be public? |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
210 if attr[:12] == '_Request__r_': |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
211 name = attr[12:] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
212 if hasattr(Request, 'get_' + name): |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
213 getattr(self, 'get_' + name)() |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
214 return getattr(self, attr) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
215 raise AttributeError, attr |
| 11669 | 216 |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
217 def get_method(self): |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
218 if self.has_data(): |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
219 return "POST" |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
220 else: |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
221 return "GET" |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
222 |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
223 # XXX these helper methods are lame |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
224 |
| 11669 | 225 def add_data(self, data): |
| 226 self.data = data | |
| 227 | |
| 228 def has_data(self): | |
| 229 return self.data is not None | |
| 230 | |
| 231 def get_data(self): | |
| 232 return self.data | |
| 233 | |
| 234 def get_full_url(self): | |
| 235 return self.__original | |
| 236 | |
| 237 def get_type(self): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
238 if self.type is None: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
239 self.type, self.__r_type = splittype(self.__original) |
|
17723
9f6cfdf71d50
Raise useful exception when called with URL for which request type
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
17528
diff
changeset
|
240 if self.type is None: |
|
9f6cfdf71d50
Raise useful exception when called with URL for which request type
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
17528
diff
changeset
|
241 raise ValueError, "unknown url type: %s" % self.__original |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
242 return self.type |
| 11669 | 243 |
| 244 def get_host(self): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
245 if self.host is None: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
246 self.host, self.__r_host = splithost(self.__r_type) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
247 if self.host: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
248 self.host = unquote(self.host) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
249 return self.host |
| 11669 | 250 |
| 251 def get_selector(self): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
252 return self.__r_host |
| 11669 | 253 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
254 def set_proxy(self, host, type): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
255 self.host, self.type = host, type |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
256 self.__r_host = self.__original |
| 11669 | 257 |
|
48245
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
258 def has_proxy(self): |
|
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
259 return self.__r_host == self.__original |
|
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
260 |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
261 def get_origin_req_host(self): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
262 return self.origin_req_host |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
263 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
264 def is_unverifiable(self): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
265 return self.unverifiable |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
266 |
| 11669 | 267 def add_header(self, key, val): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
268 # useful for something like authentication |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
269 self.headers[key.capitalize()] = val |
| 11669 | 270 |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
271 def add_unredirected_header(self, key, val): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
272 # will not be added to a redirected request |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
273 self.unredirected_hdrs[key.capitalize()] = val |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
274 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
275 def has_header(self, header_name): |
|
32291
40c524c1c0fe
Cleanup: the in operator already returns a bool, no need to bool() it again
Neal Norwitz <nnorwitz@gmail.com>
parents:
32164
diff
changeset
|
276 return (header_name in self.headers or |
|
40c524c1c0fe
Cleanup: the in operator already returns a bool, no need to bool() it again
Neal Norwitz <nnorwitz@gmail.com>
parents:
32164
diff
changeset
|
277 header_name in self.unredirected_hdrs) |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
278 |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
279 def get_header(self, header_name, default=None): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
280 return self.headers.get( |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
281 header_name, |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
282 self.unredirected_hdrs.get(header_name, default)) |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
283 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
284 def header_items(self): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
285 hdrs = self.unredirected_hdrs.copy() |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
286 hdrs.update(self.headers) |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
287 return hdrs.items() |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
288 |
| 11669 | 289 class OpenerDirector: |
| 290 def __init__(self): | |
|
35156
3c6aed3f3a31
bug [ 1175848 ] poorly named variable in urllib2.py
Georg Brandl <georg@python.org>
parents:
34545
diff
changeset
|
291 client_version = "Python-urllib/%s" % __version__ |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
292 self.addheaders = [('User-agent', client_version)] |
| 11669 | 293 # manage the individual handlers |
| 294 self.handlers = [] | |
| 295 self.handle_open = {} | |
| 296 self.handle_error = {} | |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
297 self.process_response = {} |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
298 self.process_request = {} |
| 11669 | 299 |
| 300 def add_handler(self, handler): | |
|
42211
255bd43ca78f
Patch #1752270, #1750931: complain if urllib2 add_handler called
Georg Brandl <georg@python.org>
parents:
41977
diff
changeset
|
301 if not hasattr(handler, "add_parent"): |
|
255bd43ca78f
Patch #1752270, #1750931: complain if urllib2 add_handler called
Georg Brandl <georg@python.org>
parents:
41977
diff
changeset
|
302 raise TypeError("expected BaseHandler instance, got %r" % |
|
255bd43ca78f
Patch #1752270, #1750931: complain if urllib2 add_handler called
Georg Brandl <georg@python.org>
parents:
41977
diff
changeset
|
303 type(handler)) |
|
255bd43ca78f
Patch #1752270, #1750931: complain if urllib2 add_handler called
Georg Brandl <georg@python.org>
parents:
41977
diff
changeset
|
304 |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
305 added = False |
|
20335
83837ac5bb40
Fix [ #465502 ] urllib2: urlopen unicode problem
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
19504
diff
changeset
|
306 for meth in dir(handler): |
|
38487
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
307 if meth in ["redirect_request", "do_open", "proxy_open"]: |
|
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
308 # oops, coincidental match |
|
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
309 continue |
|
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
310 |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
311 i = meth.find("_") |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
312 protocol = meth[:i] |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
313 condition = meth[i+1:] |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
314 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
315 if condition.startswith("error"): |
|
32291
40c524c1c0fe
Cleanup: the in operator already returns a bool, no need to bool() it again
Neal Norwitz <nnorwitz@gmail.com>
parents:
32164
diff
changeset
|
316 j = condition.find("_") + i + 1 |
| 11669 | 317 kind = meth[j+1:] |
| 318 try: | |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
319 kind = int(kind) |
| 11669 | 320 except ValueError: |
| 321 pass | |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
322 lookup = self.handle_error.get(protocol, {}) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
323 self.handle_error[protocol] = lookup |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
324 elif condition == "open": |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
325 kind = protocol |
| 34543 | 326 lookup = self.handle_open |
| 327 elif condition == "response": | |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
328 kind = protocol |
| 34543 | 329 lookup = self.process_response |
| 330 elif condition == "request": | |
| 331 kind = protocol | |
| 332 lookup = self.process_request | |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
333 else: |
| 11669 | 334 continue |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
335 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
336 handlers = lookup.setdefault(kind, []) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
337 if handlers: |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
338 bisect.insort(handlers, handler) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
339 else: |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
340 handlers.append(handler) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
341 added = True |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
342 |
| 11669 | 343 if added: |
|
41006
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
344 # the handlers must work in an specific order, the order |
|
e1b000bed462
Minor corrections to docs, and an explanation comentary
Facundo Batista <facundobatista@gmail.com>
parents:
40689
diff
changeset
|
345 # is specified in a Handler attribute |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
346 bisect.insort(self.handlers, handler) |
| 11669 | 347 handler.add_parent(self) |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
348 |
| 11669 | 349 def close(self): |
|
31198
d9811759e7de
Remove __del__ methods to avoid creating uncollectable cyclic trash.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31185
diff
changeset
|
350 # Only exists for backwards compatibility. |
|
d9811759e7de
Remove __del__ methods to avoid creating uncollectable cyclic trash.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31185
diff
changeset
|
351 pass |
| 11669 | 352 |
| 353 def _call_chain(self, chain, kind, meth_name, *args): | |
|
37401
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
354 # Handlers raise an exception if no one else should try to handle |
|
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
355 # the request, or return None if they can't but another handler |
|
cf10f2443542
Patch #1463012: remove not working undocumented classes from urllib2
Georg Brandl <georg@python.org>
parents:
37400
diff
changeset
|
356 # could. Otherwise, they return the response. |
| 11669 | 357 handlers = chain.get(kind, ()) |
| 358 for handler in handlers: | |
| 359 func = getattr(handler, meth_name) | |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
360 |
|
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
361 result = func(*args) |
| 11669 | 362 if result is not None: |
| 363 return result | |
| 364 | |
|
47372
e2c5ae30cffe
Fixed the semantic of timeout for socket.create_connection and
Facundo Batista <facundobatista@gmail.com>
parents:
46662
diff
changeset
|
365 def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
366 # accept a URL or a Request object |
|
23645
e1e90b0f1f45
Remove uses of the string and types modules:
Walter Dörwald <walter@livinglogic.de>
parents:
23620
diff
changeset
|
367 if isinstance(fullurl, basestring): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
368 req = Request(fullurl, data) |
| 11669 | 369 else: |
| 370 req = fullurl | |
| 371 if data is not None: | |
| 372 req.add_data(data) | |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
373 |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
374 req.timeout = timeout |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
375 protocol = req.get_type() |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
376 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
377 # pre-process request |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
378 meth_name = protocol+"_request" |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
379 for processor in self.process_request.get(protocol, []): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
380 meth = getattr(processor, meth_name) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
381 req = meth(req) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
382 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
383 response = self._open(req, data) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
384 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
385 # post-process response |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
386 meth_name = protocol+"_response" |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
387 for processor in self.process_response.get(protocol, []): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
388 meth = getattr(processor, meth_name) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
389 response = meth(req, response) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
390 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
391 return response |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
392 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
393 def _open(self, req, data=None): |
| 11669 | 394 result = self._call_chain(self.handle_open, 'default', |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
395 'default_open', req) |
| 11669 | 396 if result: |
| 397 return result | |
| 398 | |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
399 protocol = req.get_type() |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
400 result = self._call_chain(self.handle_open, protocol, protocol + |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
401 '_open', req) |
| 11669 | 402 if result: |
| 403 return result | |
| 404 | |
| 405 return self._call_chain(self.handle_open, 'unknown', | |
| 406 'unknown_open', req) | |
| 407 | |
| 408 def error(self, proto, *args): | |
|
34545
9d66fba78444
Replace list of constants with tuples of constants.
Raymond Hettinger <python@rcn.com>
parents:
34543
diff
changeset
|
409 if proto in ('http', 'https'): |
|
20916
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
410 # XXX http[s] protocols are special-cased |
|
b9850c44756b
Clean up one comment, fix typos in others.
Fred Drake <fdrake@acm.org>
parents:
20335
diff
changeset
|
411 dict = self.handle_error['http'] # https is not different than http |
| 11669 | 412 proto = args[2] # YUCK! |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
413 meth_name = 'http_error_%s' % proto |
| 11669 | 414 http_err = 1 |
| 415 orig_args = args | |
| 416 else: | |
| 417 dict = self.handle_error | |
| 418 meth_name = proto + '_error' | |
| 419 http_err = 0 | |
| 420 args = (dict, proto, meth_name) + args | |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
421 result = self._call_chain(*args) |
| 11669 | 422 if result: |
| 423 return result | |
| 424 | |
| 425 if http_err: | |
| 426 args = (dict, 'default', 'http_error_default') + orig_args | |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
427 return self._call_chain(*args) |
| 11669 | 428 |
|
29369
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
429 # XXX probably also want an abstract factory that knows when it makes |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
430 # sense to skip a superclass in favor of a subclass and when it might |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
431 # make sense to include both |
| 11669 | 432 |
| 433 def build_opener(*handlers): | |
| 434 """Create an opener object from a list of handlers. | |
| 435 | |
| 436 The opener will use several default handlers, including support | |
|
29369
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
437 for HTTP and FTP. |
| 11669 | 438 |
| 439 If any of the handlers passed as arguments are subclasses of the | |
| 440 default handlers, the default handlers will not be used. | |
| 441 """ | |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
442 import types |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
443 def isclass(obj): |
|
51540
d69769b4fd20
no need for this __bases__ trick anymore
Benjamin Peterson <benjamin@python.org>
parents:
50734
diff
changeset
|
444 return isinstance(obj, (types.ClassType, type)) |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
445 |
| 11669 | 446 opener = OpenerDirector() |
| 447 default_classes = [ProxyHandler, UnknownHandler, HTTPHandler, | |
| 448 HTTPDefaultErrorHandler, HTTPRedirectHandler, | |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
449 FTPHandler, FileHandler, HTTPErrorProcessor] |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
450 if hasattr(httplib, 'HTTPS'): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
451 default_classes.append(HTTPSHandler) |
|
46534
fed72fa967bf
Issue #2670: urllib2.build_opener() failed when two handlers
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
46371
diff
changeset
|
452 skip = set() |
| 11669 | 453 for klass in default_classes: |
| 454 for check in handlers: | |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
455 if isclass(check): |
| 11669 | 456 if issubclass(check, klass): |
|
46534
fed72fa967bf
Issue #2670: urllib2.build_opener() failed when two handlers
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
46371
diff
changeset
|
457 skip.add(klass) |
|
20335
83837ac5bb40
Fix [ #465502 ] urllib2: urlopen unicode problem
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
19504
diff
changeset
|
458 elif isinstance(check, klass): |
|
46534
fed72fa967bf
Issue #2670: urllib2.build_opener() failed when two handlers
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
46371
diff
changeset
|
459 skip.add(klass) |
| 11669 | 460 for klass in skip: |
| 461 default_classes.remove(klass) | |
| 462 | |
| 463 for klass in default_classes: | |
| 464 opener.add_handler(klass()) | |
| 465 | |
| 466 for h in handlers: | |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
467 if isclass(h): |
| 11669 | 468 h = h() |
| 469 opener.add_handler(h) | |
| 470 return opener | |
| 471 | |
| 472 class BaseHandler: | |
|
29369
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
473 handler_order = 500 |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
474 |
| 11669 | 475 def add_parent(self, parent): |
| 476 self.parent = parent | |
|
31389
aca1ce1b4b61
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
31208
diff
changeset
|
477 |
| 11669 | 478 def close(self): |
|
31198
d9811759e7de
Remove __del__ methods to avoid creating uncollectable cyclic trash.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31185
diff
changeset
|
479 # Only exists for backwards compatibility |
|
d9811759e7de
Remove __del__ methods to avoid creating uncollectable cyclic trash.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31185
diff
changeset
|
480 pass |
|
31389
aca1ce1b4b61
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
31208
diff
changeset
|
481 |
|
29369
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
482 def __lt__(self, other): |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
483 if not hasattr(other, "handler_order"): |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
484 # Try to preserve the old behavior of having custom classes |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
485 # inserted after default ones (works only for custom user |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
486 # classes which are not aware of handler_order). |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
487 return True |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
488 return self.handler_order < other.handler_order |
|
29488
5143cfe7cc75
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
29454
diff
changeset
|
489 |
| 11669 | 490 |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
491 class HTTPErrorProcessor(BaseHandler): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
492 """Process HTTP error responses.""" |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
493 handler_order = 1000 # after all other processing |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
494 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
495 def http_response(self, request, response): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
496 code, msg, hdrs = response.code, response.msg, response.info() |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
497 |
|
41532
4b9fa8092e82
Whitespace normalization. Ugh, we really need to do this more often.
Neal Norwitz <nnorwitz@gmail.com>
parents:
41514
diff
changeset
|
498 # According to RFC 2616, "2xx" code indicates that the client's |
|
41510
810d333a73a1
As specified in RFC 2616, 2xx code indicates that the client's
Facundo Batista <facundobatista@gmail.com>
parents:
41144
diff
changeset
|
499 # request was successfully received, understood, and accepted. |
|
810d333a73a1
As specified in RFC 2616, 2xx code indicates that the client's
Facundo Batista <facundobatista@gmail.com>
parents:
41144
diff
changeset
|
500 if not (200 <= code < 300): |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
501 response = self.parent.error( |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
502 'http', request, response, code, msg, hdrs) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
503 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
504 return response |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
505 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
506 https_response = http_response |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
507 |
| 11669 | 508 class HTTPDefaultErrorHandler(BaseHandler): |
| 509 def http_error_default(self, req, fp, code, msg, hdrs): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
510 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) |
| 11669 | 511 |
| 512 class HTTPRedirectHandler(BaseHandler): | |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
513 # maximum number of redirections to any single URL |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
514 # this is needed because of the state that cookies introduce |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
515 max_repeats = 4 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
516 # maximum total number of redirections (regardless of URL) before |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
517 # assuming we're in a loop |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
518 max_redirections = 10 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
519 |
|
28973
68f042fafb7d
Better fix for newurl as suggested by Jim Jewett in SF bug #730963.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28971
diff
changeset
|
520 def redirect_request(self, req, fp, code, msg, headers, newurl): |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
521 """Return a Request or None in response to a redirect. |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
522 |
|
29798
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
523 This is called by the http_error_30x methods when a |
|
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
524 redirection response is received. If a redirection should |
|
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
525 take place, return a new Request to allow http_error_30x to |
|
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
526 perform the redirect. Otherwise, raise HTTPError if no-one |
|
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
527 else should try to handle this url. Return None if you can't |
|
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
528 but another Handler might. |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
529 """ |
|
28970
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
530 m = req.get_method() |
|
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
531 if (code in (301, 302, 303, 307) and m in ("GET", "HEAD") |
|
29823
ba5f0b808192
Patch 549151, rev4: redirect posts for 301 also. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents:
29798
diff
changeset
|
532 or code in (301, 302, 303) and m == "POST"): |
|
ba5f0b808192
Patch 549151, rev4: redirect posts for 301 also. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents:
29798
diff
changeset
|
533 # Strictly (according to RFC 2616), 301 or 302 in response |
|
ba5f0b808192
Patch 549151, rev4: redirect posts for 301 also. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents:
29798
diff
changeset
|
534 # to a POST MUST NOT cause a redirection without confirmation |
|
28970
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
535 # from the user (of urllib2, in this case). In practice, |
|
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
536 # essentially all clients do redirect in this case, so we |
|
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
537 # do the same. |
|
37074
d13430a8bd2c
Bug #1353433: be conciliant with spaces in redirect URLs
Georg Brandl <georg@python.org>
parents:
36563
diff
changeset
|
538 # be conciliant with URIs containing a space |
|
d13430a8bd2c
Bug #1353433: be conciliant with spaces in redirect URLs
Georg Brandl <georg@python.org>
parents:
36563
diff
changeset
|
539 newurl = newurl.replace(' ', '%20') |
|
45218
219f7f2134d4
Fixes Issue 1401. When redirected, a possible POST get converted
Facundo Batista <facundobatista@gmail.com>
parents:
44903
diff
changeset
|
540 newheaders = dict((k,v) for k,v in req.headers.items() |
|
219f7f2134d4
Fixes Issue 1401. When redirected, a possible POST get converted
Facundo Batista <facundobatista@gmail.com>
parents:
44903
diff
changeset
|
541 if k.lower() not in ("content-length", "content-type") |
|
219f7f2134d4
Fixes Issue 1401. When redirected, a possible POST get converted
Facundo Batista <facundobatista@gmail.com>
parents:
44903
diff
changeset
|
542 ) |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
543 return Request(newurl, |
|
45218
219f7f2134d4
Fixes Issue 1401. When redirected, a possible POST get converted
Facundo Batista <facundobatista@gmail.com>
parents:
44903
diff
changeset
|
544 headers=newheaders, |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
545 origin_req_host=req.get_origin_req_host(), |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
546 unverifiable=True) |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
547 else: |
|
29454
9b72558a1eb9
Patch #754340: Fix typo in redirect_request. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents:
29369
diff
changeset
|
548 raise HTTPError(req.get_full_url(), code, msg, headers, fp) |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
549 |
| 11669 | 550 # Implementation note: To avoid the server sending us into an |
| 551 # infinite loop, the request object needs to track what URLs we | |
| 552 # have already seen. Do this by adding a handler-specific | |
| 553 # attribute to the Request object. | |
| 554 def http_error_302(self, req, fp, code, msg, headers): | |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
555 # Some servers (incorrectly) return multiple Location headers |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
556 # (so probably same goes for URI). Use first header. |
|
23620
a1de0bab5475
SF 563203. Replaced 'has_key()' with 'in'.
Raymond Hettinger <python@rcn.com>
parents:
23483
diff
changeset
|
557 if 'location' in headers: |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
558 newurl = headers.getheaders('location')[0] |
|
23620
a1de0bab5475
SF 563203. Replaced 'has_key()' with 'in'.
Raymond Hettinger <python@rcn.com>
parents:
23483
diff
changeset
|
559 elif 'uri' in headers: |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
560 newurl = headers.getheaders('uri')[0] |
| 11669 | 561 else: |
| 562 return | |
|
48259
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
563 |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
564 # fix a possible malformed URL |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
565 urlparts = urlparse.urlparse(newurl) |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
566 if not urlparts.path: |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
567 urlparts = list(urlparts) |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
568 urlparts[2] = "/" |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
569 newurl = urlparse.urlunparse(urlparts) |
|
af4e40f893bd
Issue 2464. Supports a malformation in the URL received
Facundo Batista <facundobatista@gmail.com>
parents:
48245
diff
changeset
|
570 |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
571 newurl = urlparse.urljoin(req.get_full_url(), newurl) |
|
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
572 |
| 11669 | 573 # XXX Probably want to forget about the state of the current |
| 574 # request, although that might interact poorly with other | |
| 575 # handlers that also use handler-specific request attributes | |
|
28973
68f042fafb7d
Better fix for newurl as suggested by Jim Jewett in SF bug #730963.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28971
diff
changeset
|
576 new = self.redirect_request(req, fp, code, msg, headers, newurl) |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
577 if new is None: |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
578 return |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
579 |
|
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
580 # loop detection |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
581 # .redirect_dict has a key url if url was previously visited. |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
582 if hasattr(req, 'redirect_dict'): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
583 visited = new.redirect_dict = req.redirect_dict |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
584 if (visited.get(newurl, 0) >= self.max_repeats or |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
585 len(visited) >= self.max_redirections): |
| 11669 | 586 raise HTTPError(req.get_full_url(), code, |
|
19047
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
587 self.inf_msg + msg, headers, fp) |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
588 else: |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
589 visited = new.redirect_dict = req.redirect_dict = {} |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
590 visited[newurl] = visited.get(newurl, 0) + 1 |
|
19047
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
591 |
|
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
592 # Don't close the fp until we are sure that we won't use it |
|
19119
600a9a2b7a18
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
19047
diff
changeset
|
593 # with HTTPError. |
|
19047
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
594 fp.read() |
|
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
595 fp.close() |
|
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
596 |
| 11669 | 597 return self.parent.open(new) |
| 598 | |
|
28771
29b6a48f4c45
SF Patch 549151: urllib2 POSTs on redirect
Raymond Hettinger <python@rcn.com>
parents:
26080
diff
changeset
|
599 http_error_301 = http_error_303 = http_error_307 = http_error_302 |
| 11669 | 600 |
|
29823
ba5f0b808192
Patch 549151, rev4: redirect posts for 301 also. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents:
29798
diff
changeset
|
601 inf_msg = "The HTTP server returned a redirect error that would " \ |
|
13620
5b727ab37092
Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents:
13050
diff
changeset
|
602 "lead to an infinite loop.\n" \ |
|
29823
ba5f0b808192
Patch 549151, rev4: redirect posts for 301 also. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents:
29798
diff
changeset
|
603 "The last 30x error message was:\n" |
| 11669 | 604 |
|
37400
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
605 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
606 def _parse_proxy(proxy): |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
607 """Return (scheme, user, password, host/port) given a URL or an authority. |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
608 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
609 If a URL is supplied, it must have an authority (host:port) component. |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
610 According to RFC 3986, having an authority component means the URL must |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
611 have two slashes after the scheme: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
612 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
613 >>> _parse_proxy('file:/ftp.example.com/') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
614 Traceback (most recent call last): |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
615 ValueError: proxy URL with no authority: 'file:/ftp.example.com/' |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
616 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
617 The first three items of the returned tuple may be None. |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
618 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
619 Examples of authority parsing: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
620 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
621 >>> _parse_proxy('proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
622 (None, None, None, 'proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
623 >>> _parse_proxy('proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
624 (None, None, None, 'proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
625 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
626 The authority component may optionally include userinfo (assumed to be |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
627 username:password): |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
628 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
629 >>> _parse_proxy('joe:password@proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
630 (None, 'joe', 'password', 'proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
631 >>> _parse_proxy('joe:password@proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
632 (None, 'joe', 'password', 'proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
633 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
634 Same examples, but with URLs instead: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
635 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
636 >>> _parse_proxy('http://proxy.example.com/') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
637 ('http', None, None, 'proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
638 >>> _parse_proxy('http://proxy.example.com:3128/') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
639 ('http', None, None, 'proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
640 >>> _parse_proxy('http://joe:password@proxy.example.com/') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
641 ('http', 'joe', 'password', 'proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
642 >>> _parse_proxy('http://joe:password@proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
643 ('http', 'joe', 'password', 'proxy.example.com:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
644 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
645 Everything after the authority is ignored: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
646 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
647 >>> _parse_proxy('ftp://joe:password@proxy.example.com/rubbish:3128') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
648 ('ftp', 'joe', 'password', 'proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
649 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
650 Test for no trailing '/' case: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
651 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
652 >>> _parse_proxy('http://joe:password@proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
653 ('http', 'joe', 'password', 'proxy.example.com') |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
654 |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
655 """ |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
656 scheme, r_scheme = splittype(proxy) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
657 if not r_scheme.startswith("/"): |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
658 # authority |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
659 scheme = None |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
660 authority = proxy |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
661 else: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
662 # URL |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
663 if not r_scheme.startswith("//"): |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
664 raise ValueError("proxy URL with no authority: %r" % proxy) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
665 # We have an authority, so for RFC 3986-compliant URLs (by ss 3. |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
666 # and 3.3.), path is empty or starts with '/' |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
667 end = r_scheme.find("/", 2) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
668 if end == -1: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
669 end = None |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
670 authority = r_scheme[2:end] |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
671 userinfo, hostport = splituser(authority) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
672 if userinfo is not None: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
673 user, password = splitpasswd(userinfo) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
674 else: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
675 user = password = None |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
676 return scheme, user, password, hostport |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
677 |
| 11669 | 678 class ProxyHandler(BaseHandler): |
|
29369
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
679 # Proxies must be in front |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
680 handler_order = 100 |
|
58c1ad417ee7
- urllib2.py now knows how to order proxy classes, so the user doesn't
Gustavo Niemeyer <gustavo@niemeyer.net>
parents:
29243
diff
changeset
|
681 |
| 11669 | 682 def __init__(self, proxies=None): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
683 if proxies is None: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
684 proxies = getproxies() |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
685 assert hasattr(proxies, 'has_key'), "proxies must be a mapping" |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
686 self.proxies = proxies |
|
29162
384bda343725
To be on the safe side, backed out any questionable iteritem changes and set back to item calls.
Brett Cannon <bcannon@gmail.com>
parents:
29154
diff
changeset
|
687 for type, url in proxies.items(): |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
688 setattr(self, '%s_open' % type, |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
689 lambda r, proxy=url, type=type, meth=self.proxy_open: \ |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
690 meth(r, proxy, type)) |
| 11669 | 691 |
| 692 def proxy_open(self, req, proxy, type): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
693 orig_type = req.get_type() |
|
37400
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
694 proxy_type, user, password, hostport = _parse_proxy(proxy) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
695 if proxy_type is None: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
696 proxy_type = orig_type |
|
36364
97885adf73b5
Bug #902075: urllib2 now handles "host:port" proxy specifications
Georg Brandl <georg@python.org>
parents:
35465
diff
changeset
|
697 if user and password: |
|
37400
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
698 user_pass = '%s:%s' % (unquote(user), unquote(password)) |
|
40205
9d691d3755d0
[Patch #1574068 by Scott Dial] urllib and urllib2 were using
Andrew M. Kuchling <amk@amk.ca>
parents:
39595
diff
changeset
|
699 creds = base64.b64encode(user_pass).strip() |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
700 req.add_header('Proxy-authorization', 'Basic ' + creds) |
|
37400
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
701 hostport = unquote(hostport) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
702 req.set_proxy(hostport, proxy_type) |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
703 if orig_type == proxy_type: |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
704 # let other handlers take care of it |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
705 return None |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
706 else: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
707 # need to start over, because the other handlers don't |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
708 # grok the proxy's URL type |
|
37400
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
709 # e.g. if we have a constructor arg proxies like so: |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
710 # {'http': 'ftp://proxy.example.com'}, we may end up turning |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
711 # a request for http://acme.example.com/a into one for |
|
4837722a60f0
Patch #1462790: fix urllib2 ProxyHandler for host:port proxies
Georg Brandl <georg@python.org>
parents:
37399
diff
changeset
|
712 # ftp://proxy.example.com/a |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
713 return self.parent.open(req) |
| 11669 | 714 |
| 715 class HTTPPasswordMgr: | |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
716 |
| 11669 | 717 def __init__(self): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
718 self.passwd = {} |
| 11669 | 719 |
| 720 def add_password(self, realm, uri, user, passwd): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
721 # uri could be a single URI or a sequence |
|
23645
e1e90b0f1f45
Remove uses of the string and types modules:
Walter Dörwald <walter@livinglogic.de>
parents:
23620
diff
changeset
|
722 if isinstance(uri, basestring): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
723 uri = [uri] |
|
23620
a1de0bab5475
SF 563203. Replaced 'has_key()' with 'in'.
Raymond Hettinger <python@rcn.com>
parents:
23483
diff
changeset
|
724 if not realm in self.passwd: |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
725 self.passwd[realm] = {} |
|
38466
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
726 for default_port in True, False: |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
727 reduced_uri = tuple( |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
728 [self.reduce_uri(u, default_port) for u in uri]) |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
729 self.passwd[realm][reduced_uri] = (user, passwd) |
| 11669 | 730 |
| 731 def find_user_password(self, realm, authuri): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
732 domains = self.passwd.get(realm, {}) |
|
38466
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
733 for default_port in True, False: |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
734 reduced_authuri = self.reduce_uri(authuri, default_port) |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
735 for uris, authinfo in domains.iteritems(): |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
736 for uri in uris: |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
737 if self.is_suburi(uri, reduced_authuri): |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
738 return authinfo |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
739 return None, None |
| 11669 | 740 |
|
38466
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
741 def reduce_uri(self, uri, default_port=True): |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
742 """Accept authority or URI and extract only the authority and path.""" |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
743 # note HTTP URLs do not have a userinfo component |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
744 parts = urlparse.urlsplit(uri) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
745 if parts[1]: |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
746 # URI |
|
38466
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
747 scheme = parts[0] |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
748 authority = parts[1] |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
749 path = parts[2] or '/' |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
750 else: |
|
38466
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
751 # host or host:port |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
752 scheme = None |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
753 authority = uri |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
754 path = '/' |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
755 host, port = splitport(authority) |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
756 if default_port and port is None and scheme is not None: |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
757 dport = {"http": 80, |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
758 "https": 443, |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
759 }.get(scheme) |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
760 if dport is not None: |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
761 authority = "%s:%d" % (host, dport) |
|
71861c378093
Patch #1496206: urllib2 PasswordMgr ./. default ports
Georg Brandl <georg@python.org>
parents:
38211
diff
changeset
|
762 return authority, path |
| 11669 | 763 |
| 764 def is_suburi(self, base, test): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
765 """Check if test is below base in a URI tree |
| 11669 | 766 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
767 Both args must be URIs in reduced form. |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
768 """ |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
769 if base == test: |
|
22867
e2f2187c3d2c
Partial introduction of bools where appropriate.
Guido van Rossum <guido@python.org>
parents:
22368
diff
changeset
|
770 return True |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
771 if base[0] != test[0]: |
|
22867
e2f2187c3d2c
Partial introduction of bools where appropriate.
Guido van Rossum <guido@python.org>
parents:
22368
diff
changeset
|
772 return False |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
773 common = posixpath.commonprefix((base[1], test[1])) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
774 if len(common) == len(base[1]): |
|
22867
e2f2187c3d2c
Partial introduction of bools where appropriate.
Guido van Rossum <guido@python.org>
parents:
22368
diff
changeset
|
775 return True |
|
e2f2187c3d2c
Partial introduction of bools where appropriate.
Guido van Rossum <guido@python.org>
parents:
22368
diff
changeset
|
776 return False |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
777 |
| 11669 | 778 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
779 class HTTPPasswordMgrWithDefaultRealm(HTTPPasswordMgr): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
780 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
781 def find_user_password(self, realm, authuri): |
|
29798
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
782 user, password = HTTPPasswordMgr.find_user_password(self, realm, |
|
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
783 authuri) |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
784 if user is not None: |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
785 return user, password |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
786 return HTTPPasswordMgr.find_user_password(self, None, authuri) |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
787 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
788 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
789 class AbstractBasicAuthHandler: |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
790 |
|
41048
242de16a6d44
Patch #812285: allow multiple auth schemes in AbstractBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
41006
diff
changeset
|
791 # XXX this allows for multiple auth-schemes, but will stupidly pick |
|
242de16a6d44
Patch #812285: allow multiple auth schemes in AbstractBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
41006
diff
changeset
|
792 # the last one with a realm specified. |
| 11669 | 793 |
|
45980
7f035592d6e1
#2136: allow single quotes in realm spec.
Georg Brandl <georg@python.org>
parents:
45218
diff
changeset
|
794 # allow for double- and single-quoted realm values |
|
7f035592d6e1
#2136: allow single quotes in realm spec.
Georg Brandl <georg@python.org>
parents:
45218
diff
changeset
|
795 # (single quotes are a violation of the RFC, but appear in the wild) |
|
7f035592d6e1
#2136: allow single quotes in realm spec.
Georg Brandl <georg@python.org>
parents:
45218
diff
changeset
|
796 rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+' |
|
7f035592d6e1
#2136: allow single quotes in realm spec.
Georg Brandl <georg@python.org>
parents:
45218
diff
changeset
|
797 'realm=(["\'])(.*?)\\2', re.I) |
| 11669 | 798 |
|
38487
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
799 # XXX could pre-emptively send auth info already accepted (RFC 2617, |
|
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
800 # end of section 2, and section 1.2 immediately after "credentials" |
|
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
801 # production). |
|
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
802 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
803 def __init__(self, password_mgr=None): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
804 if password_mgr is None: |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
805 password_mgr = HTTPPasswordMgr() |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
806 self.passwd = password_mgr |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
807 self.add_password = self.passwd.add_password |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
808 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
809 def http_error_auth_reqed(self, authreq, host, req, headers): |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
810 # host may be an authority (without userinfo) or a URL with an |
|
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
811 # authority |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
812 # XXX could be multiple headers |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
813 authreq = headers.get(authreq, None) |
| 11669 | 814 if authreq: |
|
32926
5f58fcd7accf
Patch #994595: Recognize Basic auth even if other schemes are offered.
Martin v. Löwis <martin@v.loewis.de>
parents:
32620
diff
changeset
|
815 mo = AbstractBasicAuthHandler.rx.search(authreq) |
| 11669 | 816 if mo: |
|
45980
7f035592d6e1
#2136: allow single quotes in realm spec.
Georg Brandl <georg@python.org>
parents:
45218
diff
changeset
|
817 scheme, quote, realm = mo.groups() |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
818 if scheme.lower() == 'basic': |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
819 return self.retry_http_basic_auth(host, req, realm) |
| 11669 | 820 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
821 def retry_http_basic_auth(self, host, req, realm): |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
822 user, pw = self.passwd.find_user_password(realm, host) |
|
31990
cbbacda84480
Patch #944110: Properly process empty passwords. Fixes #944082.
Martin v. Löwis <martin@v.loewis.de>
parents:
31877
diff
changeset
|
823 if pw is not None: |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
824 raw = "%s:%s" % (user, pw) |
|
40205
9d691d3755d0
[Patch #1574068 by Scott Dial] urllib and urllib2 were using
Andrew M. Kuchling <amk@amk.ca>
parents:
39595
diff
changeset
|
825 auth = 'Basic %s' % base64.b64encode(raw).strip() |
|
20926
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
826 if req.headers.get(self.auth_header, None) == auth: |
|
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
827 return None |
|
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
828 req.add_header(self.auth_header, auth) |
|
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
829 return self.parent.open(req) |
| 11669 | 830 else: |
| 831 return None | |
| 832 | |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
833 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
834 class HTTPBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
835 |
|
20926
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
836 auth_header = 'Authorization' |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
837 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
838 def http_error_401(self, req, fp, code, msg, headers): |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
839 url = req.get_full_url() |
|
17055
743296ce052b
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
16895
diff
changeset
|
840 return self.http_error_auth_reqed('www-authenticate', |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
841 url, req, headers) |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
842 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
843 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
844 class ProxyBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler): |
| 11669 | 845 |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
846 auth_header = 'Proxy-authorization' |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
847 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
848 def http_error_407(self, req, fp, code, msg, headers): |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
849 # http_error_auth_reqed requires that there is no userinfo component in |
|
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
850 # authority. Assume there isn't one, since urllib2 does not (and |
|
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
851 # should not, RFC 3986 s. 3.2.1) support requests for URLs containing |
|
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
852 # userinfo. |
|
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
853 authority = req.get_host() |
|
17055
743296ce052b
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
16895
diff
changeset
|
854 return self.http_error_auth_reqed('proxy-authenticate', |
|
38051
2a28915616d5
Patch #1470846: fix urllib2 ProxyBasicAuthHandler.
Georg Brandl <georg@python.org>
parents:
37403
diff
changeset
|
855 authority, req, headers) |
| 11669 | 856 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
857 |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
858 def randombytes(n): |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
859 """Return n random bytes.""" |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
860 # Use /dev/urandom if it is available. Fall back to random module |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
861 # if not. It might be worthwhile to extend this function to use |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
862 # other platform-specific mechanisms for getting random bytes. |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
863 if os.path.exists("/dev/urandom"): |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
864 f = open("/dev/urandom") |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
865 s = f.read(n) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
866 f.close() |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
867 return s |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
868 else: |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
869 L = [chr(random.randrange(0, 256)) for i in range(n)] |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
870 return "".join(L) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
871 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
872 class AbstractDigestAuthHandler: |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
873 # Digest authentication is specified in RFC 2617. |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
874 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
875 # XXX The client does not inspect the Authentication-Info header |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
876 # in a successful response. |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
877 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
878 # XXX It should be possible to test this implementation against |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
879 # a mock server that just generates a static set of challenges. |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
880 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
881 # XXX qop="auth-int" supports is shaky |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
882 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
883 def __init__(self, passwd=None): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
884 if passwd is None: |
|
19047
35050d188a6a
Fix SF bug [ #447370 ] typo in urllib2.py
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
18936
diff
changeset
|
885 passwd = HTTPPasswordMgr() |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
886 self.passwd = passwd |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
887 self.add_password = self.passwd.add_password |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
888 self.retried = 0 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
889 self.nonce_count = 0 |
| 11669 | 890 |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
891 def reset_retry_count(self): |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
892 self.retried = 0 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
893 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
894 def http_error_auth_reqed(self, auth_header, host, req, headers): |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
895 authreq = headers.get(auth_header, None) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
896 if self.retried > 5: |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
897 # Don't fail endlessly - if we failed once, we'll probably |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
898 # fail a second time. Hm. Unless the Password Manager is |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
899 # prompting for the information. Crap. This isn't great |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
900 # but it's better than the current 'repeat until recursion |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
901 # depth exceeded' approach <wink> |
|
31389
aca1ce1b4b61
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
31208
diff
changeset
|
902 raise HTTPError(req.get_full_url(), 401, "digest auth failed", |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
903 headers, None) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
904 else: |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
905 self.retried += 1 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
906 if authreq: |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
907 scheme = authreq.split()[0] |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
908 if scheme.lower() == 'digest': |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
909 return self.retry_http_digest_auth(req, authreq) |
| 11669 | 910 |
| 911 def retry_http_digest_auth(self, req, auth): | |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
912 token, challenge = auth.split(' ', 1) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
913 chal = parse_keqv_list(parse_http_list(challenge)) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
914 auth = self.get_authorization(req, chal) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
915 if auth: |
|
20926
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
916 auth_val = 'Digest %s' % auth |
|
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
917 if req.headers.get(self.auth_header, None) == auth_val: |
|
7cedbfac746b
Fix SF bug 468948 & 451295: urllib2 authentication problems
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
20916
diff
changeset
|
918 return None |
|
38098
23d4ac33ac16
Patch #1480067: don't redirect HTTP digest auth in urllib2
Georg Brandl <georg@python.org>
parents:
38052
diff
changeset
|
919 req.add_unredirected_header(self.auth_header, auth_val) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
920 resp = self.parent.open(req) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
921 return resp |
| 11669 | 922 |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
923 def get_cnonce(self, nonce): |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
924 # The cnonce-value is an opaque |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
925 # quoted string value provided by the client and used by both client |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
926 # and server to avoid chosen plaintext attacks, to provide mutual |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
927 # authentication, and to provide some message integrity protection. |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
928 # This isn't a fabulous effort, but it's probably Good Enough. |
|
38052
32a187f3d5ba
In stdlib, use hashlib instead of deprecated md5 and sha modules.
Georg Brandl <georg@python.org>
parents:
38051
diff
changeset
|
929 dig = hashlib.sha1("%s:%s:%s:%s" % (self.nonce_count, nonce, time.ctime(), |
|
32a187f3d5ba
In stdlib, use hashlib instead of deprecated md5 and sha modules.
Georg Brandl <georg@python.org>
parents:
38051
diff
changeset
|
930 randombytes(8))).hexdigest() |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
931 return dig[:16] |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
932 |
| 11669 | 933 def get_authorization(self, req, chal): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
934 try: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
935 realm = chal['realm'] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
936 nonce = chal['nonce'] |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
937 qop = chal.get('qop') |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
938 algorithm = chal.get('algorithm', 'MD5') |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
939 # mod_digest doesn't send an opaque, even though it isn't |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
940 # supposed to be optional |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
941 opaque = chal.get('opaque', None) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
942 except KeyError: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
943 return None |
| 11669 | 944 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
945 H, KD = self.get_algorithm_impls(algorithm) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
946 if H is None: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
947 return None |
| 11669 | 948 |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
949 user, pw = self.passwd.find_user_password(realm, req.get_full_url()) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
950 if user is None: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
951 return None |
| 11669 | 952 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
953 # XXX not implemented yet |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
954 if req.has_data(): |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
955 entdig = self.get_entity_digest(req.get_data(), chal) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
956 else: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
957 entdig = None |
| 11669 | 958 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
959 A1 = "%s:%s:%s" % (user, realm, pw) |
|
34376
2b485974b471
Patch #1095362: replace hardcoded test for POST/GET with call to get_method,
Johannes Gijsbers <jlg@dds.nl>
parents:
34227
diff
changeset
|
960 A2 = "%s:%s" % (req.get_method(), |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
961 # XXX selector: what about proxies and full urls |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
962 req.get_selector()) |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
963 if qop == 'auth': |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
964 self.nonce_count += 1 |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
965 ncvalue = '%08x' % self.nonce_count |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
966 cnonce = self.get_cnonce(nonce) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
967 noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, H(A2)) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
968 respdig = KD(H(A1), noncebit) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
969 elif qop is None: |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
970 respdig = KD(H(A1), "%s:%s" % (nonce, H(A2))) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
971 else: |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
972 # XXX handle auth-int. |
|
41977
823c2f7591f5
Patch #1667860: Fix UnboundLocalError in urllib2.
Georg Brandl <georg@python.org>
parents:
41967
diff
changeset
|
973 raise URLError("qop '%s' is not supported." % qop) |
|
31389
aca1ce1b4b61
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
31208
diff
changeset
|
974 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
975 # XXX should the partial digests be encoded too? |
| 11669 | 976 |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
977 base = 'username="%s", realm="%s", nonce="%s", uri="%s", ' \ |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
978 'response="%s"' % (user, realm, nonce, req.get_selector(), |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
979 respdig) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
980 if opaque: |
|
34227
e2088ae0d67f
Two small changes, SF bug #974757 and SF patch #1037974.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
33716
diff
changeset
|
981 base += ', opaque="%s"' % opaque |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
982 if entdig: |
|
34227
e2088ae0d67f
Two small changes, SF bug #974757 and SF patch #1037974.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
33716
diff
changeset
|
983 base += ', digest="%s"' % entdig |
|
e2088ae0d67f
Two small changes, SF bug #974757 and SF patch #1037974.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
33716
diff
changeset
|
984 base += ', algorithm="%s"' % algorithm |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
985 if qop: |
|
34227
e2088ae0d67f
Two small changes, SF bug #974757 and SF patch #1037974.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
33716
diff
changeset
|
986 base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
987 return base |
| 11669 | 988 |
| 989 def get_algorithm_impls(self, algorithm): | |
|
46662
7172e11e9fc8
#2695: Do case-insensitive check for algorithms.
Georg Brandl <georg@python.org>
parents:
46534
diff
changeset
|
990 # algorithm should be case-insensitive according to RFC2617 |
|
7172e11e9fc8
#2695: Do case-insensitive check for algorithms.
Georg Brandl <georg@python.org>
parents:
46534
diff
changeset
|
991 algorithm = algorithm.upper() |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
992 # lambdas assume digest modules are imported at the top level |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
993 if algorithm == 'MD5': |
|
38052
32a187f3d5ba
In stdlib, use hashlib instead of deprecated md5 and sha modules.
Georg Brandl <georg@python.org>
parents:
38051
diff
changeset
|
994 H = lambda x: hashlib.md5(x).hexdigest() |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
995 elif algorithm == 'SHA': |
|
38052
32a187f3d5ba
In stdlib, use hashlib instead of deprecated md5 and sha modules.
Georg Brandl <georg@python.org>
parents:
38051
diff
changeset
|
996 H = lambda x: hashlib.sha1(x).hexdigest() |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
997 # XXX MD5-sess |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
998 KD = lambda s, d: H("%s:%s" % (s, d)) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
999 return H, KD |
| 11669 | 1000 |
| 1001 def get_entity_digest(self, data, chal): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1002 # XXX not implemented yet |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1003 return None |
| 11669 | 1004 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1005 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1006 class HTTPDigestAuthHandler(BaseHandler, AbstractDigestAuthHandler): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1007 """An authentication protocol defined by RFC 2069 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1008 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1009 Digest authentication improves on basic authentication because it |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1010 does not transmit passwords in the clear. |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1011 """ |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1012 |
|
29798
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
1013 auth_header = 'Authorization' |
|
38487
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
1014 handler_order = 490 # before Basic auth |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1015 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1016 def http_error_401(self, req, fp, code, msg, headers): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1017 host = urlparse.urlparse(req.get_full_url())[1] |
|
31389
aca1ce1b4b61
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
31208
diff
changeset
|
1018 retry = self.http_error_auth_reqed('www-authenticate', |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
1019 host, req, headers) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
1020 self.reset_retry_count() |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
1021 return retry |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1022 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1023 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1024 class ProxyDigestAuthHandler(BaseHandler, AbstractDigestAuthHandler): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1025 |
|
29798
57b93c94830a
Fix wrong header name. The framework looks for auth_header.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
29521
diff
changeset
|
1026 auth_header = 'Proxy-Authorization' |
|
38487
5a12c0c5183e
Patches #1497027 and #972322: try HTTP digest auth first,
Georg Brandl <georg@python.org>
parents:
38466
diff
changeset
|
1027 handler_order = 490 # before Basic auth |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1028 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1029 def http_error_407(self, req, fp, code, msg, headers): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1030 host = req.get_host() |
|
31389
aca1ce1b4b61
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
31208
diff
changeset
|
1031 retry = self.http_error_auth_reqed('proxy-authenticate', |
|
30740
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
1032 host, req, headers) |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
1033 self.reset_retry_count() |
|
494d5cf0bbe3
Apply patch 823328 -- support for rfc 2617 digestion authentication.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30722
diff
changeset
|
1034 return retry |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
1035 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1036 class AbstractHTTPHandler(BaseHandler): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1037 |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1038 def __init__(self, debuglevel=0): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1039 self._debuglevel = debuglevel |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1040 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1041 def set_http_debuglevel(self, level): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1042 self._debuglevel = level |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1043 |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1044 def do_request_(self, request): |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1045 host = request.get_host() |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1046 if not host: |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1047 raise URLError('no host given') |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1048 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1049 if request.has_data(): # POST |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1050 data = request.get_data() |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1051 if not request.has_header('Content-type'): |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1052 request.add_unredirected_header( |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1053 'Content-type', |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1054 'application/x-www-form-urlencoded') |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1055 if not request.has_header('Content-length'): |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1056 request.add_unredirected_header( |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1057 'Content-length', '%d' % len(data)) |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1058 |
|
48245
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
1059 sel_host = host |
|
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
1060 if request.has_proxy(): |
|
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
1061 scheme, sel = splittype(request.get_selector()) |
|
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
1062 sel_host, sel_path = splithost(sel) |
|
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
1063 |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1064 if not request.has_header('Host'): |
|
48245
1702873f2492
Issue #2776: fixed small issue when handling an URL with double slash
Facundo Batista <facundobatista@gmail.com>
parents:
47372
diff
changeset
|
1065 request.add_unredirected_header('Host', sel_host) |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1066 for name, value in self.parent.addheaders: |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1067 name = name.capitalize() |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1068 if not request.has_header(name): |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1069 request.add_unredirected_header(name, value) |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1070 |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1071 return request |
|
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1072 |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1073 def do_open(self, http_class, req): |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1074 """Return an addinfourl object for the request, using http_class. |
|
28970
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
1075 |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1076 http_class must implement the HTTPConnection API from httplib. |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1077 The addinfourl return value is a file-like object. It also |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1078 has methods and attributes including: |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1079 - info(): return a mimetools.Message object for the headers |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1080 - geturl(): return the original request URL |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1081 - code: HTTP status code |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1082 """ |
|
17403
34ec5e2fa356
Idiotic braino caused HTTP openers to ignore proxies.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
17082
diff
changeset
|
1083 host = req.get_host() |
| 11669 | 1084 if not host: |
| 1085 raise URLError('no host given') | |
| 1086 | |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1087 h = http_class(host, timeout=req.timeout) # will parse host:port |
|
31185
74c154708b0c
SF patch 852995: add processors feature to urllib2
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
30740
diff
changeset
|
1088 h.set_debuglevel(self._debuglevel) |
|
15874
4c01ee0a8073
Whitespace normalization. Top level of Lib now fixed-point for reindent.py!
Tim Peters <tim.peters@gmail.com>
parents:
15179
diff
changeset
|
1089 |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1090 headers = dict(req.headers) |
|
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1091 headers.update(req.unredirected_hdrs) |
|
31581
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1092 # We want to make an HTTP/1.1 request, but the addinfourl |
|
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1093 # class isn't prepared to deal with a persistent connection. |
|
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1094 # It will try to read all remaining data from the socket, |
|
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1095 # which will block while the server waits for the next request. |
|
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1096 # So make sure the connection gets closed after the (only) |
|
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1097 # request. |
|
5e7455fb8db6
Fix two bugs in the new do_open() implementation for HTTPHandler.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31536
diff
changeset
|
1098 headers["Connection"] = "close" |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1099 headers = dict( |
|
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1100 (name.title(), val) for name, val in headers.items()) |
|
28970
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
1101 try: |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1102 h.request(req.get_method(), req.get_selector(), req.data, headers) |
|
50734
d09d6fe31b61
Issue 4879: Allow buffering for HTTPResponse
Kristján Valur Jónsson <kristjan@ccpgames.com>
parents:
48278
diff
changeset
|
1103 try: |
|
d09d6fe31b61
Issue 4879: Allow buffering for HTTPResponse
Kristján Valur Jónsson <kristjan@ccpgames.com>
parents:
48278
diff
changeset
|
1104 r = h.getresponse(buffering=True) |
|
d09d6fe31b61
Issue 4879: Allow buffering for HTTPResponse
Kristján Valur Jónsson <kristjan@ccpgames.com>
parents:
48278
diff
changeset
|
1105 except TypeError: #buffering kw not supported |
|
d09d6fe31b61
Issue 4879: Allow buffering for HTTPResponse
Kristján Valur Jónsson <kristjan@ccpgames.com>
parents:
48278
diff
changeset
|
1106 r = h.getresponse() |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1107 except socket.error, err: # XXX what error? |
|
28970
9f9022c21056
Repair redirect handling and raise URLError on host-not-found.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
28771
diff
changeset
|
1108 raise URLError(err) |
| 11669 | 1109 |
|
32579
492dc8b8e58e
[Patch #988504] Fix HTTP error handling via a patch from John J. Lee
Andrew M. Kuchling <amk@amk.ca>
parents:
32442
diff
changeset
|
1110 # Pick apart the HTTPResponse object to get the addinfourl |
|
33010
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1111 # object initialized properly. |
|
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1112 |
|
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1113 # Wrap the HTTPResponse object in socket's file object adapter |
|
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1114 # for Windows. That adapter calls recv(), so delegate recv() |
|
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1115 # to read(). This weird wrapping allows the returned object to |
|
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1116 # have readline() and readlines() methods. |
|
33039
420488408e0e
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
33010
diff
changeset
|
1117 |
|
33010
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1118 # XXX It might be better to extract the read buffering code |
|
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1119 # out of socket._fileobject() and into a base class. |
|
33039
420488408e0e
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
33010
diff
changeset
|
1120 |
|
33010
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1121 r.recv = r.read |
|
40684
2fa6e82b1f17
Patch #1627441: close sockets properly in urllib2.
Georg Brandl <georg@python.org>
parents:
40205
diff
changeset
|
1122 fp = socket._fileobject(r, close=True) |
|
33039
420488408e0e
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
33010
diff
changeset
|
1123 |
|
33010
8f6d205d531b
Fix urllib2.urlopen() handling of chunked content encoding.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
32926
diff
changeset
|
1124 resp = addinfourl(fp, r.msg, req.get_full_url()) |
|
32579
492dc8b8e58e
[Patch #988504] Fix HTTP error handling via a patch from John J. Lee
Andrew M. Kuchling <amk@amk.ca>
parents:
32442
diff
changeset
|
1125 resp.code = r.status |
|
492dc8b8e58e
[Patch #988504] Fix HTTP error handling via a patch from John J. Lee
Andrew M. Kuchling <amk@amk.ca>
parents:
32442
diff
changeset
|
1126 resp.msg = r.reason |
|
492dc8b8e58e
[Patch #988504] Fix HTTP error handling via a patch from John J. Lee
Andrew M. Kuchling <amk@amk.ca>
parents:
32442
diff
changeset
|
1127 return resp |
| 11669 | 1128 |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1129 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1130 class HTTPHandler(AbstractHTTPHandler): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1131 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1132 def http_open(self, req): |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1133 return self.do_open(httplib.HTTPConnection, req) |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1134 |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1135 http_request = AbstractHTTPHandler.do_request_ |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1136 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1137 if hasattr(httplib, 'HTTPS'): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1138 class HTTPSHandler(AbstractHTTPHandler): |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1139 |
|
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1140 def https_open(self, req): |
|
31208
9493dd222fbb
Rewrite AbstractHTTPHandler to use modern httplib interface.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
31198
diff
changeset
|
1141 return self.do_open(httplib.HTTPSConnection, req) |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1142 |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1143 https_request = AbstractHTTPHandler.do_request_ |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1144 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1145 class HTTPCookieProcessor(BaseHandler): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1146 def __init__(self, cookiejar=None): |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
1147 import cookielib |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1148 if cookiejar is None: |
|
32291
40c524c1c0fe
Cleanup: the in operator already returns a bool, no need to bool() it again
Neal Norwitz <nnorwitz@gmail.com>
parents:
32164
diff
changeset
|
1149 cookiejar = cookielib.CookieJar() |
|
32164
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1150 self.cookiejar = cookiejar |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1151 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1152 def http_request(self, request): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1153 self.cookiejar.add_cookie_header(request) |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1154 return request |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1155 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1156 def http_response(self, request, response): |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1157 self.cookiejar.extract_cookies(response, request) |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1158 return response |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1159 |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1160 https_request = http_request |
|
18d6a8f99823
Patch #963318: Add support for client-side cookie management.
Martin v. Löwis <martin@v.loewis.de>
parents:
32034
diff
changeset
|
1161 https_response = http_response |
|
16895
8aecdb223488
Checking in patch 404826 -- urllib2 enhancements and documentations.
Moshe Zadka <moshez@math.huji.ac.il>
parents:
16514
diff
changeset
|
1162 |
| 11669 | 1163 class UnknownHandler(BaseHandler): |
| 1164 def unknown_open(self, req): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1165 type = req.get_type() |
| 11669 | 1166 raise URLError('unknown url type: %s' % type) |
| 1167 | |
| 1168 def parse_keqv_list(l): | |
| 1169 """Parse list of key=value strings where keys are not duplicated.""" | |
| 1170 parsed = {} | |
| 1171 for elt in l: | |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
1172 k, v = elt.split('=', 1) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1173 if v[0] == '"' and v[-1] == '"': |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1174 v = v[1:-1] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1175 parsed[k] = v |
| 11669 | 1176 return parsed |
| 1177 | |
| 1178 def parse_http_list(s): | |
| 1179 """Parse lists as described by RFC 2068 Section 2. | |
|
35465
3e5c07faab11
Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents:
35429
diff
changeset
|
1180 |
| 31877 | 1181 In particular, parse comma-separated lists where the elements of |
| 11669 | 1182 the list may include quoted-strings. A quoted-string could |
|
35429
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1183 contain a comma. A non-quoted string could have quotes in the |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1184 middle. Neither commas nor quotes count if they are escaped. |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1185 Only double-quotes count, not single-quotes. |
| 11669 | 1186 """ |
|
35429
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1187 res = [] |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1188 part = '' |
| 11669 | 1189 |
|
35429
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1190 escape = quote = False |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1191 for cur in s: |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1192 if escape: |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1193 part += cur |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1194 escape = False |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1195 continue |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1196 if quote: |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1197 if cur == '\\': |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1198 escape = True |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1199 continue |
|
35429
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1200 elif cur == '"': |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1201 quote = False |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1202 part += cur |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1203 continue |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1204 |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1205 if cur == ',': |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1206 res.append(part) |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1207 part = '' |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1208 continue |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1209 |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1210 if cur == '"': |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1211 quote = True |
|
35465
3e5c07faab11
Whitespace normalization (via reindent.py).
Tim Peters <tim.peters@gmail.com>
parents:
35429
diff
changeset
|
1212 |
|
35429
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1213 part += cur |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1214 |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1215 # append last part |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1216 if part: |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1217 res.append(part) |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1218 |
|
e0bd6f324fd6
Bug #735248: Fix urllib2.parse_http_list.
Georg Brandl <georg@python.org>
parents:
35426
diff
changeset
|
1219 return [part.strip() for part in res] |
| 11669 | 1220 |
| 1221 class FileHandler(BaseHandler): | |
| 1222 # Use local file or FTP depending on form of URL | |
| 1223 def file_open(self, req): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1224 url = req.get_selector() |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1225 if url[:2] == '//' and url[2:3] != '/': |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1226 req.type = 'ftp' |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1227 return self.parent.open(req) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1228 else: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1229 return self.open_local_file(req) |
| 11669 | 1230 |
| 1231 # names for the localhost | |
| 1232 names = None | |
| 1233 def get_names(self): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1234 if FileHandler.names is None: |
|
37399
f4d88cfaec60
bug #1462706: guard against host not having FQDN hostname
Georg Brandl <georg@python.org>
parents:
37351
diff
changeset
|
1235 try: |
|
f4d88cfaec60
bug #1462706: guard against host not having FQDN hostname
Georg Brandl <georg@python.org>
parents:
37351
diff
changeset
|
1236 FileHandler.names = (socket.gethostbyname('localhost'), |
|
f4d88cfaec60
bug #1462706: guard against host not having FQDN hostname
Georg Brandl <georg@python.org>
parents:
37351
diff
changeset
|
1237 socket.gethostbyname(socket.gethostname())) |
|
f4d88cfaec60
bug #1462706: guard against host not having FQDN hostname
Georg Brandl <georg@python.org>
parents:
37351
diff
changeset
|
1238 except socket.gaierror: |
|
f4d88cfaec60
bug #1462706: guard against host not having FQDN hostname
Georg Brandl <georg@python.org>
parents:
37351
diff
changeset
|
1239 FileHandler.names = (socket.gethostbyname('localhost'),) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1240 return FileHandler.names |
| 11669 | 1241 |
| 1242 # not entirely sure what the rules are here | |
| 1243 def open_local_file(self, req): | |
|
40689
b23711be2b3e
Use new email module names (#1637162, #1637159, #1637157).
Georg Brandl <georg@python.org>
parents:
40684
diff
changeset
|
1244 import email.utils |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
1245 import mimetypes |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1246 host = req.get_host() |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1247 file = req.get_selector() |
|
19504
59931db572b2
Add content-type header to ftp URLs (SF patch #454553)
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
19466
diff
changeset
|
1248 localfile = url2pathname(file) |
|
41144
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1249 try: |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1250 stats = os.stat(localfile) |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1251 size = stats.st_size |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1252 modified = email.utils.formatdate(stats.st_mtime, usegmt=True) |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1253 mtype = mimetypes.guess_type(file)[0] |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1254 headers = mimetools.Message(StringIO( |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1255 'Content-type: %s\nContent-length: %d\nLast-modified: %s\n' % |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1256 (mtype or 'text/plain', size, modified))) |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1257 if host: |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1258 host, port = splitport(host) |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1259 if not host or \ |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1260 (not port and socket.gethostbyname(host) in self.get_names()): |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1261 return addinfourl(open(localfile, 'rb'), |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1262 headers, 'file:'+file) |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1263 except OSError, msg: |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1264 # urllib2 users shouldn't expect OSErrors coming from urlopen() |
|
7a331c7f9404
Patch #1668100: urllib2 now correctly raises URLError instead of
Georg Brandl <georg@python.org>
parents:
41133
diff
changeset
|
1265 raise URLError(msg) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1266 raise URLError('file not on local host') |
| 11669 | 1267 |
| 1268 class FTPHandler(BaseHandler): | |
| 1269 def ftp_open(self, req): | |
|
38205
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
1270 import ftplib |
|
0b4731049b71
Delay-import some large modules to speed up urllib2 import.
Georg Brandl <georg@python.org>
parents:
38132
diff
changeset
|
1271 import mimetypes |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1272 host = req.get_host() |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1273 if not host: |
|
44903
22a9d9757a16
Fix the test_urllib2net failures that were caused by r58067.
Neal Norwitz <nnorwitz@gmail.com>
parents:
43299
diff
changeset
|
1274 raise URLError('ftp error: no host given') |
|
31536
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1275 host, port = splitport(host) |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1276 if port is None: |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1277 port = ftplib.FTP_PORT |
|
32620
2fd69240729a
Patch [ 972332 ] urllib2 FTPHandler bugs / John J. Lee
Kurt B. Kaiser <kbk@shore.net>
parents:
32594
diff
changeset
|
1278 else: |
|
2fd69240729a
Patch [ 972332 ] urllib2 FTPHandler bugs / John J. Lee
Kurt B. Kaiser <kbk@shore.net>
parents:
32594
diff
changeset
|
1279 port = int(port) |
|
31536
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1280 |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1281 # username/password handling |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1282 user, host = splituser(host) |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1283 if user: |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1284 user, passwd = splitpasswd(user) |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1285 else: |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1286 passwd = None |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1287 host = unquote(host) |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1288 user = unquote(user or '') |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1289 passwd = unquote(passwd or '') |
|
7c052be40169
Patch #711838: Allow non-anonymous ftp urls in urllib2.
Martin v. Löwis <martin@v.loewis.de>
parents:
31531
diff
changeset
|
1290 |
|
15179
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
1291 try: |
|
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
1292 host = socket.gethostbyname(host) |
|
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
1293 except socket.error, msg: |
|
015da13f0038
two fixes for redirects:
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
13620
diff
changeset
|
1294 raise URLError(msg) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1295 path, attrs = splitattr(req.get_selector()) |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
1296 dirs = path.split('/') |
|
31531
78362aaa491d
Patch #817379: Allow for absolute ftp paths.
Martin v. Löwis <martin@v.loewis.de>
parents:
31389
diff
changeset
|
1297 dirs = map(unquote, dirs) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1298 dirs, file = dirs[:-1], dirs[-1] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1299 if dirs and not dirs[0]: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1300 dirs = dirs[1:] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1301 try: |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1302 fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1303 type = file and 'I' or 'D' |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1304 for attr in attrs: |
|
32620
2fd69240729a
Patch [ 972332 ] urllib2 FTPHandler bugs / John J. Lee
Kurt B. Kaiser <kbk@shore.net>
parents:
32594
diff
changeset
|
1305 attr, value = splitvalue(attr) |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
1306 if attr.lower() == 'type' and \ |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1307 value in ('a', 'A', 'i', 'I', 'd', 'D'): |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15874
diff
changeset
|
1308 type = value.upper() |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1309 fp, retrlen = fw.retrfile(file, type) |
|
19466
61d53c053e43
SF patch #454553 by Walter Dörwald: add content-type to FTP URLs, like
Guido van Rossum <guido@python.org>
parents:
19176
diff
changeset
|
1310 headers = "" |
|
61d53c053e43
SF patch #454553 by Walter Dörwald: add content-type to FTP URLs, like
Guido van Rossum <guido@python.org>
parents:
19176
diff
changeset
|
1311 mtype = mimetypes.guess_type(req.get_full_url())[0] |
|
61d53c053e43
SF patch #454553 by Walter Dörwald: add content-type to FTP URLs, like
Guido van Rossum <guido@python.org>
parents:
19176
diff
changeset
|
1312 if mtype: |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1313 headers += "Content-type: %s\n" % mtype |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1314 if retrlen is not None and retrlen >= 0: |
|
39595
44622eb58a25
Patch #1542948: fix urllib2 header casing issue. With new test.
Georg Brandl <georg@python.org>
parents:
39188
diff
changeset
|
1315 headers += "Content-length: %d\n" % retrlen |
|
19466
61d53c053e43
SF patch #454553 by Walter Dörwald: add content-type to FTP URLs, like
Guido van Rossum <guido@python.org>
parents:
19176
diff
changeset
|
1316 sf = StringIO(headers) |
|
61d53c053e43
SF patch #454553 by Walter Dörwald: add content-type to FTP URLs, like
Guido van Rossum <guido@python.org>
parents:
19176
diff
changeset
|
1317 headers = mimetools.Message(sf) |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1318 return addinfourl(fp, headers, req.get_full_url()) |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1319 except ftplib.all_errors, msg: |
|
44903
22a9d9757a16
Fix the test_urllib2net failures that were caused by r58067.
Neal Norwitz <nnorwitz@gmail.com>
parents:
43299
diff
changeset
|
1320 raise URLError, ('ftp error: %s' % msg), sys.exc_info()[2] |
| 11669 | 1321 |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1322 def connect_ftp(self, user, passwd, host, port, dirs, timeout): |
|
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1323 fw = ftpwrapper(user, passwd, host, port, dirs, timeout) |
| 11669 | 1324 ## fw.ftp.set_debuglevel(1) |
| 1325 return fw | |
| 1326 | |
| 1327 class CacheFTPHandler(FTPHandler): | |
| 1328 # XXX would be nice to have pluggable cache strategies | |
| 1329 # XXX this stuff is definitely not thread safe | |
| 1330 def __init__(self): | |
| 1331 self.cache = {} | |
| 1332 self.timeout = {} | |
| 1333 self.soonest = 0 | |
| 1334 self.delay = 60 | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1335 self.max_conns = 16 |
| 11669 | 1336 |
| 1337 def setTimeout(self, t): | |
| 1338 self.delay = t | |
| 1339 | |
| 1340 def setMaxConns(self, m): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1341 self.max_conns = m |
| 11669 | 1342 |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1343 def connect_ftp(self, user, passwd, host, port, dirs, timeout): |
|
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1344 key = user, host, port, '/'.join(dirs), timeout |
|
23620
a1de0bab5475
SF 563203. Replaced 'has_key()' with 'in'.
Raymond Hettinger <python@rcn.com>
parents:
23483
diff
changeset
|
1345 if key in self.cache: |
| 11669 | 1346 self.timeout[key] = time.time() + self.delay |
| 1347 else: | |
|
41967
f141c5f77860
Added an optional timeout parameter to function urllib2.urlopen,
Facundo Batista <facundobatista@gmail.com>
parents:
41743
diff
changeset
|
1348 self.cache[key] = ftpwrapper(user, passwd, host, port, dirs, timeout) |
| 11669 | 1349 self.timeout[key] = time.time() + self.delay |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1350 self.check_cache() |
| 11669 | 1351 return self.cache[key] |
| 1352 | |
| 1353 def check_cache(self): | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1354 # first check for old ones |
| 11669 | 1355 t = time.time() |
| 1356 if self.soonest <= t: | |
|
29243
64227f8de7da
iteritems() should not have been used for self.timeout which changes during the loop.
Raymond Hettinger <python@rcn.com>
parents:
29162
diff
changeset
|
1357 for k, v in self.timeout.items(): |
| 11669 | 1358 if v < t: |
| 1359 self.cache[k].close() | |
| 1360 del self.cache[k] | |
| 1361 del self.timeout[k] | |
| 1362 self.soonest = min(self.timeout.values()) | |
| 1363 | |
| 1364 # then check the size | |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1365 if len(self.cache) == self.max_conns: |
|
29154
5e6d0dcf155f
Revert some changes back to dict.items made in a previous patch.
Brett Cannon <bcannon@gmail.com>
parents:
29080
diff
changeset
|
1366 for k, v in self.timeout.items(): |
|
11742
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1367 if v == self.soonest: |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1368 del self.cache[k] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1369 del self.timeout[k] |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1370 break |
|
840e18eec07f
Untabify to pass the -tt test.
Fred Drake <fdrake@acm.org>
parents:
11707
diff
changeset
|
1371 self.soonest = min(self.timeout.values()) |