cpython-lru_cache-weakref: Lib/urllib.py annotate

annotate Lib/urllib.py @ 79675:c0a2294de787 2.7

Issue #16199: Backport 2to3 typo fix from 3.2.

author	Chris Jerdonek <chris.jerdonek@gmail.com>
date	Thu, 11 Oct 2012 16:15:46 -0700
parents	6ce4868861ba
children	4927899bea8d

rev	line source
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1 """Open an arbitrary URL.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	2
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	3 See the following document for more info on URLs:
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	4 "Names and Addresses, URIs, URLs, URNs, URCs", at
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	5 http://www.w3.org/pub/WWW/Addressing/Overview.html
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	6
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	7 See also the HTTP spec (from which the error codes are derived):
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	8 "HTTP - Hypertext Transfer Protocol", at
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	9 http://www.w3.org/pub/WWW/Protocols/
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	10
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	11 Related standards and specs:
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	12 - RFC1808: the "relative URL" spec. (authoritative status)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	13 - RFC1738 - the "URL standard". (authoritative status)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	14 - RFC1630 - the "URI spec". (informational status)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	15
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	16 The object returned by URLopener().open(file) will differ per
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	17 protocol. All you know is that is has methods read(), readline(),
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	18 readlines(), fileno(), close() and info(). The read*(), fileno()
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	19 and close() methods work like those of open files.
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	20 The info() method returns a mimetools.Message object which can be
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	21 used to query various info about the object, if available.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	22 (mimetools.Message objects are queried with the getheader() method.)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	23 """
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	24
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	25 import string
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	26 import socket
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	27 import os
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	28 import time
4558 df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	29 import sys
74321 01ef13e9e225 - Issue #13642: Unquote before b64encoding user:password during Basic Senthil Kumaran <senthil@uthcode.com> parents: 74033 diff changeset	30 import base64
01ef13e9e225 - Issue #13642: Unquote before b64encoding user:password during Basic Senthil Kumaran <senthil@uthcode.com> parents: 74033 diff changeset	31
31788 2024b4e6e38d Replace code in urllib for basejoin (undocumented) with urlparse.urljoin . Brett Cannon <bcannon@gmail.com> parents: 31515 diff changeset	32 from urlparse import urljoin as basejoin
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	33
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16833 diff changeset	34 __all__ = ["urlopen", "URLopener", "FancyURLopener", "urlretrieve",
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16833 diff changeset	35 "urlcleanup", "quote", "quote_plus", "unquote", "unquote_plus",
17039 5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	36 "urlencode", "url2pathname", "pathname2url", "splittag",
5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	37 "localhost", "thishost", "ftperrors", "basejoin", "unwrap",
5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	38 "splittype", "splithost", "splituser", "splitpasswd", "splitport",
5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	39 "splitnport", "splitquery", "splitattr", "splitvalue",
41743 2ecc296443c3 Remove the gopherlib module. It has been raising a DeprecationWarning since Brett Cannon <bcannon@gmail.com> parents: 41246 diff changeset	40 "getproxies"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16833 diff changeset	41
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	42 __version__ = '1.17' # XXX This version is not always updated :-(
5727 e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	43
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	44 MAXFTPCACHE = 10 # Trim the ftp cache beyond this size
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	45
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	46 # Helper for non-unix systems
61258 f6785bce54b5 In a number of places code still revers Ronald Oussoren <ronaldoussoren@mac.com> parents: 61138 diff changeset	47 if os.name == 'nt':
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	48 from nturl2path import url2pathname, pathname2url
16949 09ba15ee1961 RISCOS changes by dschwertberger. Guido van Rossum <guido@python.org> parents: 16889 diff changeset	49 elif os.name == 'riscos':
09ba15ee1961 RISCOS changes by dschwertberger. Guido van Rossum <guido@python.org> parents: 16889 diff changeset	50 from rourl2path import url2pathname, pathname2url
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	51 else:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	52 def url2pathname(pathname):
36124 536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	53 """OS-specific conversion from a relative URL of the 'file' scheme
536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	54 to a file system path; not recommended for general use."""
10383 70dd91c21513 From: Sjoerd Mullender Guido van Rossum <guido@python.org> parents: 10366 diff changeset	55 return unquote(pathname)
36124 536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	56
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	57 def pathname2url(pathname):
36124 536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	58 """OS-specific conversion from a file system path to a relative URL
536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	59 of the 'file' scheme; not recommended for general use."""
10383 70dd91c21513 From: Sjoerd Mullender Guido van Rossum <guido@python.org> parents: 10366 diff changeset	60 return quote(pathname)
9649 47012c13efe4 Sjoerd Mullender: Guido van Rossum <guido@python.org> parents: 9148 diff changeset	61
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	62 # This really consists of two pieces:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	63 # (1) a class which handles opening of all sorts of URLs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	64 # (plus assorted utilities etc.)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	65 # (2) a set of functions for parsing URLs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	66 # XXX Should these be separated out into different modules?
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	67
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	68
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	69 # Shortcut for basic usage
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	70 _urlopener = None
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	71 def urlopen(url, data=None, proxies=None):
47871 801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	72 """Create a file-like object for the specified URL to read from."""
801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	73 from warnings import warnpy3k
58795 ca67c37cd26d Merged revisions 78035,78040,78043,78049-78050,78052-78054 via svnmerge from Georg Brandl <georg@python.org> parents: 56582 diff changeset	74 warnpy3k("urllib.urlopen() has been removed in Python 3.0 in "
ca67c37cd26d Merged revisions 78035,78040,78043,78049-78050,78052-78054 via svnmerge from Georg Brandl <georg@python.org> parents: 56582 diff changeset	75 "favor of urllib2.urlopen()", stacklevel=2)
47871 801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	76
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	77 global _urlopener
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	78 if proxies is not None:
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	79 opener = FancyURLopener(proxies=proxies)
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	80 elif not _urlopener:
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	81 opener = FancyURLopener()
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	82 _urlopener = opener
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	83 else:
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	84 opener = _urlopener
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	85 if data is None:
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	86 return opener.open(url)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	87 else:
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	88 return opener.open(url, data)
14151 72d8a3b203b1 Randall Hopper <aa8vb@yahoo.com>>: Fred Drake <fdrake@acm.org> parents: 14118 diff changeset	89 def urlretrieve(url, filename=None, reporthook=None, data=None):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	90 global _urlopener
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	91 if not _urlopener:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	92 _urlopener = FancyURLopener()
14151 72d8a3b203b1 Randall Hopper <aa8vb@yahoo.com>>: Fred Drake <fdrake@acm.org> parents: 14118 diff changeset	93 return _urlopener.retrieve(url, filename, reporthook, data)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	94 def urlcleanup():
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	95 if _urlopener:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	96 _urlopener.cleanup()
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	97 _safe_quoters.clear()
63892 29af9abc76ca Merged revisions 81053,81259,81265 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 63035 diff changeset	98 ftpcache.clear()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	99
43008 4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	100 # check for SSL
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	101 try:
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	102 import ssl
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	103 except:
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	104 _have_ssl = False
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	105 else:
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	106 _have_ssl = True
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	107
35425 d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	108 # exception raised when downloaded size does not match content-length
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	109 class ContentTooShortError(IOError):
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	110 def __init__(self, message, content):
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	111 IOError.__init__(self, message)
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	112 self.content = content
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	113
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	114 ftpcache = {}
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	115 class URLopener:
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	116 """Class to open URLs.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	117 This is a class rather than just a subroutine because we may need
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	118 more than one set of global protocol-specific options.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	119 Note -- this is a base class for those who don't want the
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	120 automatic handling of errors type 302 (relocated) and 401
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	121 (authorization needed)."""
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	122
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	123 __tempfiles = None
4566 6403c94de50d Added safeguard against failure in __del__. Guido van Rossum <guido@python.org> parents: 4558 diff changeset	124
14157 e7c8b598ff75 Promote the server version from a local variable to a class variable, Guido van Rossum <guido@python.org> parents: 14151 diff changeset	125 version = "Python-urllib/%s" % __version__
e7c8b598ff75 Promote the server version from a local variable to a class variable, Guido van Rossum <guido@python.org> parents: 14151 diff changeset	126
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	127 # Constructor
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	128 def __init__(self, proxies=None, **x509):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	129 if proxies is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	130 proxies = getproxies()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	131 assert hasattr(proxies, 'has_key'), "proxies must be a mapping"
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	132 self.proxies = proxies
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	133 self.key_file = x509.get('key_file')
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	134 self.cert_file = x509.get('cert_file')
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	135 self.addheaders = [('User-Agent', self.version)]
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	136 self.__tempfiles = []
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	137 self.__unlink = os.unlink # See cleanup()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	138 self.tempcache = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	139 # Undocumented feature: if you assign {} to tempcache,
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	140 # it is used to cache files retrieved with
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	141 # self.retrieve(). This is not enabled by default
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	142 # since it does not work for changing documents (and I
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	143 # haven't got the logic to check expiration headers
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	144 # yet).
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	145 self.ftpcache = ftpcache
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	146 # Undocumented feature: you can use a different
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	147 # ftp cache by assigning to the .ftpcache member;
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	148 # in case you want logically independent URL openers
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	149 # XXX This is not threadsafe. Bah.
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	150
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	151 def __del__(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	152 self.close()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	153
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	154 def close(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	155 self.cleanup()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	156
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	157 def cleanup(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	158 # This code sometimes runs when the rest of this module
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	159 # has already been deleted, so it can't use any globals
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	160 # or import anything.
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	161 if self.__tempfiles:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	162 for file in self.__tempfiles:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	163 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	164 self.__unlink(file)
19172 cb346baedad6 Only catch the errors that can actually occur, as reported in bug #411881. Martin v. Löwis <martin@v.loewis.de> parents: 19119 diff changeset	165 except OSError:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	166 pass
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	167 del self.__tempfiles[:]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	168 if self.tempcache:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	169 self.tempcache.clear()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	170
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	171 def addheader(self, *args):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	172 """Add a header to be used by the HTTP interface only
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	173 e.g. u.addheader('Accept', 'sound/basic')"""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	174 self.addheaders.append(args)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	175
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	176 # External interface
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	177 def open(self, fullurl, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	178 """Use URLopener().open(file) instead of open(file, 'r')."""
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	179 fullurl = unwrap(toBytes(fullurl))
53297 c8a75710ab25 Fix for the Issue918368 - urllib doesn't correct server returned urls Senthil Kumaran <orsenthil@gmail.com> parents: 52476 diff changeset	180 # percent encode url, fixing lame server errors for e.g, like space
c8a75710ab25 Fix for the Issue918368 - urllib doesn't correct server returned urls Senthil Kumaran <orsenthil@gmail.com> parents: 52476 diff changeset	181 # within url paths.
58968 ab21a0481d72 Merged revisions 78268 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 58795 diff changeset	182 fullurl = quote(fullurl, safe="%/:=&?~#+!$,;'@()*[]\|")
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	183 if self.tempcache and fullurl in self.tempcache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	184 filename, headers = self.tempcache[fullurl]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	185 fp = open(filename, 'rb')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	186 return addinfourl(fp, headers, fullurl)
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	187 urltype, url = splittype(fullurl)
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	188 if not urltype:
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	189 urltype = 'file'
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	190 if urltype in self.proxies:
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	191 proxy = self.proxies[urltype]
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	192 urltype, proxyhost = splittype(proxy)
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	193 host, selector = splithost(proxyhost)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	194 url = (host, fullurl) # Signal special case to open_*()
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	195 else:
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	196 proxy = None
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	197 name = 'open_' + urltype
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	198 self.type = urltype
31794 f1d44295ebf7 Replace sequential split/join calls on strings with a single replace call. Brett Cannon <bcannon@gmail.com> parents: 31788 diff changeset	199 name = name.replace('-', '_')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	200 if not hasattr(self, name):
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	201 if proxy:
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	202 return self.open_unknown_proxy(proxy, fullurl, data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	203 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	204 return self.open_unknown(fullurl, data)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	205 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	206 if data is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	207 return getattr(self, name)(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	208 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	209 return getattr(self, name)(url, data)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	210 except socket.error, msg:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	211 raise IOError, ('socket error', msg), sys.exc_info()[2]
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	212
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	213 def open_unknown(self, fullurl, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	214 """Overridable interface to open unknown URL type."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	215 type, url = splittype(fullurl)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	216 raise IOError, ('url error', 'unknown url type', type)
3028 7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	217
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	218 def open_unknown_proxy(self, proxy, fullurl, data=None):
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	219 """Overridable interface to open unknown URL type."""
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	220 type, url = splittype(fullurl)
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	221 raise IOError, ('url error', 'invalid proxy for %s' % type, proxy)
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	222
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	223 # External interface
14181 89f8153fd1ba Pass data on to retrieve method. Sjoerd Mullender <sjoerd@acm.org> parents: 14157 diff changeset	224 def retrieve(self, url, filename=None, reporthook=None, data=None):
28757 99e82dbaf0cc Fix docstring for URLOpener.retrieve() in regards to opening a local file Brett Cannon <bcannon@gmail.com> parents: 28239 diff changeset	225 """retrieve(url) returns (filename, headers) for a local object
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	226 or (tempfilename, headers) for a remote object."""
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	227 url = unwrap(toBytes(url))
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	228 if self.tempcache and url in self.tempcache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	229 return self.tempcache[url]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	230 type, url1 = splittype(url)
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	231 if filename is None and (not type or type == 'file'):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	232 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	233 fp = self.open_local_file(url1)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	234 hdrs = fp.info()
57544 262a2ceaf05f actually close files Philip Jenvey <pjenvey@underboss.org> parents: 56581 diff changeset	235 fp.close()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	236 return url2pathname(splithost(url1)[1]), hdrs
58761 cc367c077245 No need to assign the results of expressions used only for side effects. Georg Brandl <georg@python.org> parents: 58759 diff changeset	237 except IOError:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	238 pass
14151 72d8a3b203b1 Randall Hopper <aa8vb@yahoo.com>>: Fred Drake <fdrake@acm.org> parents: 14118 diff changeset	239 fp = self.open(url, data)
52266 752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	240 try:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	241 headers = fp.info()
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	242 if filename:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	243 tfp = open(filename, 'wb')
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	244 else:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	245 import tempfile
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	246 garbage, path = splittype(url)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	247 garbage, path = splithost(path or "")
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	248 path, garbage = splitquery(path or "")
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	249 path, garbage = splitattr(path or "")
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	250 suffix = os.path.splitext(path)[1]
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	251 (fd, filename) = tempfile.mkstemp(suffix)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	252 self.__tempfiles.append(filename)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	253 tfp = os.fdopen(fd, 'wb')
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	254 try:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	255 result = filename, headers
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	256 if self.tempcache is not None:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	257 self.tempcache[url] = result
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	258 bs = 1024*8
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	259 size = -1
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	260 read = 0
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	261 blocknum = 0
73262 1ef30d6429db Port to 2.7 - issue 10817 - Fix urlretrieve function to raise Senthil Kumaran <senthil@uthcode.com> parents: 71764 diff changeset	262 if "content-length" in headers:
1ef30d6429db Port to 2.7 - issue 10817 - Fix urlretrieve function to raise Senthil Kumaran <senthil@uthcode.com> parents: 71764 diff changeset	263 size = int(headers["Content-Length"])
52266 752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	264 if reporthook:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	265 reporthook(blocknum, bs, size)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	266 while 1:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	267 block = fp.read(bs)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	268 if block == "":
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	269 break
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	270 read += len(block)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	271 tfp.write(block)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	272 blocknum += 1
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	273 if reporthook:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	274 reporthook(blocknum, bs, size)
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	275 finally:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	276 tfp.close()
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	277 finally:
752f2e3e4757 Merged revisions 70521 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 49260 diff changeset	278 fp.close()
35425 d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	279
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	280 # raise exception if actual size does not match content-length header
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	281 if size >= 0 and read < size:
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	282 raise ContentTooShortError("retrieval incomplete: got only %i out "
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	283 "of %i bytes" % (read, size), result)
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	284
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	285 return result
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	286
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	287 # Each method named open_<type> knows how to open that type of URL
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	288
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	289 def open_http(self, url, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	290 """Use HTTP protocol."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	291 import httplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	292 user_passwd = None
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	293 proxy_passwd= None
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	294 if isinstance(url, str):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	295 host, selector = splithost(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	296 if host:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	297 user_passwd, host = splituser(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	298 host = unquote(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	299 realhost = host
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	300 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	301 host, selector = url
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	302 # check whether the proxy contains authorization information
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	303 proxy_passwd, host = splituser(host)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	304 # now we proceed with the url we want to obtain
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	305 urltype, rest = splittype(selector)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	306 url = rest
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	307 user_passwd = None
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	308 if urltype.lower() != 'http':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	309 realhost = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	310 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	311 realhost, rest = splithost(rest)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	312 if realhost:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	313 user_passwd, realhost = splituser(realhost)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	314 if user_passwd:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	315 selector = "%s://%s%s" % (urltype, realhost, rest)
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	316 if proxy_bypass(realhost):
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	317 host = realhost
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	318
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	319 #print "proxy via http:", host, selector
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	320 if not host: raise IOError, ('http error', 'no host given')
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	321
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	322 if proxy_passwd:
74321 01ef13e9e225 - Issue #13642: Unquote before b64encoding user:password during Basic Senthil Kumaran <senthil@uthcode.com> parents: 74033 diff changeset	323 proxy_passwd = unquote(proxy_passwd)
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	324 proxy_auth = base64.b64encode(proxy_passwd).strip()
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	325 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	326 proxy_auth = None
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	327
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	328 if user_passwd:
74321 01ef13e9e225 - Issue #13642: Unquote before b64encoding user:password during Basic Senthil Kumaran <senthil@uthcode.com> parents: 74033 diff changeset	329 user_passwd = unquote(user_passwd)
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	330 auth = base64.b64encode(user_passwd).strip()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	331 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	332 auth = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	333 h = httplib.HTTP(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	334 if data is not None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	335 h.putrequest('POST', selector)
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	336 h.putheader('Content-Type', 'application/x-www-form-urlencoded')
7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	337 h.putheader('Content-Length', '%d' % len(data))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	338 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	339 h.putrequest('GET', selector)
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	340 if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' % proxy_auth)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	341 if auth: h.putheader('Authorization', 'Basic %s' % auth)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	342 if realhost: h.putheader('Host', realhost)
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	343 for args in self.addheaders: h.putheader(*args)
50673 f1ca5dd59aca Issue 4336: Let users of HTTPConnection.endheaders() submit a message body to the function if required. Kristján Valur Jónsson <kristjan@ccpgames.com> parents: 49260 diff changeset	344 h.endheaders(data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	345 errcode, errmsg, headers = h.getreply()
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	346 fp = h.getfile()
41187 90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	347 if errcode == -1:
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	348 if fp: fp.close()
41187 90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	349 # something went wrong with the HTTP status line
90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	350 raise IOError, ('http protocol error', 0,
90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	351 'got a bad status line', None)
43404 d1fa820b2dca issue1177: Ported Facundo's from urllib2 to urllib, accepting 2xx responses. Sean Reifscheider <jafo@tummy.com> parents: 43008 diff changeset	352 # According to RFC 2616, "2xx" code indicates that the client's
d1fa820b2dca issue1177: Ported Facundo's from urllib2 to urllib, accepting 2xx responses. Sean Reifscheider <jafo@tummy.com> parents: 43008 diff changeset	353 # request was successfully received, understood, and accepted.
44423 099a17e2d2bd Issue1177 Kurt B. Kaiser <kbk@shore.net> parents: 43433 diff changeset	354 if (200 <= errcode < 300):
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	355 return addinfourl(fp, headers, "http:" + url, errcode)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	356 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	357 if data is None:
10366 65e7212df1bc open_http also had the 'data is None' test backwards. don't call with the Guido van Rossum <guido@python.org> parents: 10308 diff changeset	358 return self.http_error(url, fp, errcode, errmsg, headers)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	359 else:
10366 65e7212df1bc open_http also had the 'data is None' test backwards. don't call with the Guido van Rossum <guido@python.org> parents: 10308 diff changeset	360 return self.http_error(url, fp, errcode, errmsg, headers, data)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	361
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	362 def http_error(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	363 """Handle http errors.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	364 Derived class can override this, or provide specific handlers
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	365 named http_error_DDD where DDD is the 3-digit error code."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	366 # First check if there's a specific handler for this error
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	367 name = 'http_error_%d' % errcode
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	368 if hasattr(self, name):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	369 method = getattr(self, name)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	370 if data is None:
10308 e30893961398 http_error had the 'data is None' test backwards. don't call with the Jeremy Hylton <jeremy@alum.mit.edu> parents: 10307 diff changeset	371 result = method(url, fp, errcode, errmsg, headers)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	372 else:
10308 e30893961398 http_error had the 'data is None' test backwards. don't call with the Jeremy Hylton <jeremy@alum.mit.edu> parents: 10307 diff changeset	373 result = method(url, fp, errcode, errmsg, headers, data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	374 if result: return result
10308 e30893961398 http_error had the 'data is None' test backwards. don't call with the Jeremy Hylton <jeremy@alum.mit.edu> parents: 10307 diff changeset	375 return self.http_error_default(url, fp, errcode, errmsg, headers)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	376
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	377 def http_error_default(self, url, fp, errcode, errmsg, headers):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	378 """Default error handler: close the connection and raise IOError."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	379 fp.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	380 raise IOError, ('http error', errcode, errmsg, headers)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	381
43008 4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	382 if _have_ssl:
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	383 def open_https(self, url, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	384 """Use HTTPS protocol."""
43008 4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	385
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	386 import httplib
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	387 user_passwd = None
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	388 proxy_passwd = None
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	389 if isinstance(url, str):
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	390 host, selector = splithost(url)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	391 if host:
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	392 user_passwd, host = splituser(host)
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	393 host = unquote(host)
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	394 realhost = host
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	395 else:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	396 host, selector = url
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	397 # here, we determine, whether the proxy contains authorization information
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	398 proxy_passwd, host = splituser(host)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	399 urltype, rest = splittype(selector)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	400 url = rest
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	401 user_passwd = None
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	402 if urltype.lower() != 'https':
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	403 realhost = None
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	404 else:
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	405 realhost, rest = splithost(rest)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	406 if realhost:
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	407 user_passwd, realhost = splituser(realhost)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	408 if user_passwd:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	409 selector = "%s://%s%s" % (urltype, realhost, rest)
12905 25431972294b Comment out an apparent debug print Andrew M. Kuchling <amk@amk.ca> parents: 12741 diff changeset	410 #print "proxy via https:", host, selector
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	411 if not host: raise IOError, ('https error', 'no host given')
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	412 if proxy_passwd:
74321 01ef13e9e225 - Issue #13642: Unquote before b64encoding user:password during Basic Senthil Kumaran <senthil@uthcode.com> parents: 74033 diff changeset	413 proxy_passwd = unquote(proxy_passwd)
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	414 proxy_auth = base64.b64encode(proxy_passwd).strip()
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	415 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	416 proxy_auth = None
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	417 if user_passwd:
74321 01ef13e9e225 - Issue #13642: Unquote before b64encoding user:password during Basic Senthil Kumaran <senthil@uthcode.com> parents: 74033 diff changeset	418 user_passwd = unquote(user_passwd)
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	419 auth = base64.b64encode(user_passwd).strip()
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	420 else:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	421 auth = None
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	422 h = httplib.HTTPS(host, 0,
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	423 key_file=self.key_file,
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	424 cert_file=self.cert_file)
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	425 if data is not None:
144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	426 h.putrequest('POST', selector)
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	427 h.putheader('Content-Type',
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	428 'application/x-www-form-urlencoded')
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	429 h.putheader('Content-Length', '%d' % len(data))
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	430 else:
144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	431 h.putrequest('GET', selector)
40476 dcc40b93bfbe [Patch #1617413 from Dug Song] Fix HTTP Basic authentication via HTTPS Andrew M. Kuchling <amk@amk.ca> parents: 40205 diff changeset	432 if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' % proxy_auth)
dcc40b93bfbe [Patch #1617413 from Dug Song] Fix HTTP Basic authentication via HTTPS Andrew M. Kuchling <amk@amk.ca> parents: 40205 diff changeset	433 if auth: h.putheader('Authorization', 'Basic %s' % auth)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	434 if realhost: h.putheader('Host', realhost)
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	435 for args in self.addheaders: h.putheader(*args)
50673 f1ca5dd59aca Issue 4336: Let users of HTTPConnection.endheaders() submit a message body to the function if required. Kristján Valur Jónsson <kristjan@ccpgames.com> parents: 49260 diff changeset	436 h.endheaders(data)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	437 errcode, errmsg, headers = h.getreply()
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	438 fp = h.getfile()
41187 90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	439 if errcode == -1:
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	440 if fp: fp.close()
41187 90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	441 # something went wrong with the HTTP status line
90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	442 raise IOError, ('http protocol error', 0,
90e57e2c7a03 Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41117 diff changeset	443 'got a bad status line', None)
43433 773e976bab3a #1177: accept 2xx responses for https too, not only http. Georg Brandl <georg@python.org> parents: 43404 diff changeset	444 # According to RFC 2616, "2xx" code indicates that the client's
773e976bab3a #1177: accept 2xx responses for https too, not only http. Georg Brandl <georg@python.org> parents: 43404 diff changeset	445 # request was successfully received, understood, and accepted.
44423 099a17e2d2bd Issue1177 Kurt B. Kaiser <kbk@shore.net> parents: 43433 diff changeset	446 if (200 <= errcode < 300):
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	447 return addinfourl(fp, headers, "https:" + url, errcode)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	448 else:
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	449 if data is None:
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	450 return self.http_error(url, fp, errcode, errmsg, headers)
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	451 else:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	452 return self.http_error(url, fp, errcode, errmsg, headers,
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	453 data)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	454
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	455 def open_file(self, url):
37546 edc6a0cc8466 Fix a couple of strings that were no-ops. urllib.open_file was a docstring Neal Norwitz <nnorwitz@gmail.com> parents: 37287 diff changeset	456 """Use local file or FTP depending on form of URL."""
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	457 if not isinstance(url, str):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	458 raise IOError, ('file error', 'proxy support for file protocol currently not implemented')
25240 c8044123d110 When testing for localhost/ first map to lower case. Spotted by Skip. Jack Jansen <jack.jansen@cwi.nl> parents: 25239 diff changeset	459 if url[:2] == '//' and url[2:3] != '/' and url[2:12].lower() != 'localhost/':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	460 return self.open_ftp(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	461 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	462 return self.open_local_file(url)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	463
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	464 def open_local_file(self, url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	465 """Use local file."""
40689 b23711be2b3e Use new email module names (#1637162, #1637159, #1637157). Georg Brandl <georg@python.org> parents: 40476 diff changeset	466 import mimetypes, mimetools, email.utils
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	467 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	468 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	469 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	470 from StringIO import StringIO
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	471 host, file = splithost(url)
3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	472 localname = url2pathname(file)
23034 53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	473 try:
53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	474 stats = os.stat(localname)
53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	475 except OSError, e:
53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	476 raise IOError(e.errno, e.strerror, e.filename)
22458 68fe2b391159 use stat attributes instead of tuple entries Walter Dörwald <walter@livinglogic.de> parents: 22401 diff changeset	477 size = stats.st_size
40689 b23711be2b3e Use new email module names (#1637162, #1637159, #1637157). Georg Brandl <georg@python.org> parents: 40476 diff changeset	478 modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	479 mtype = mimetypes.guess_type(url)[0]
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	480 headers = mimetools.Message(StringIO(
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	481 'Content-Type: %s\nContent-Length: %d\nLast-modified: %s\n' %
3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	482 (mtype or 'text/plain', size, modified)))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	483 if not host:
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	484 urlfile = file
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	485 if file[:1] == '/':
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	486 urlfile = 'file://' + file
74546 f6008e936fbc Fix Issue6631 - Disallow relative files paths in urllib.open() Senthil Kumaran <senthil@uthcode.com>* parents: 74321 diff changeset	487 elif file[:2] == './':
f6008e936fbc Fix Issue6631 - Disallow relative files paths in urllib.open() Senthil Kumaran <senthil@uthcode.com>* parents: 74321 diff changeset	488 raise ValueError("local file url may start with / or file:. Unknown url of type: %s" % url)
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	489 return addinfourl(open(localname, 'rb'),
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	490 headers, urlfile)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	491 host, port = splitport(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	492 if not port \
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	493 and socket.gethostbyname(host) in (localhost(), thishost()):
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	494 urlfile = file
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	495 if file[:1] == '/':
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	496 urlfile = 'file://' + file
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	497 return addinfourl(open(localname, 'rb'),
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	498 headers, urlfile)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	499 raise IOError, ('local file error', 'not on local host')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	500
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	501 def open_ftp(self, url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	502 """Use FTP protocol."""
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	503 if not isinstance(url, str):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	504 raise IOError, ('ftp error', 'proxy support for ftp protocol currently not implemented')
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	505 import mimetypes, mimetools
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	506 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	507 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	508 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	509 from StringIO import StringIO
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	510 host, path = splithost(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	511 if not host: raise IOError, ('ftp error', 'no host given')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	512 host, port = splitport(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	513 user, host = splituser(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	514 if user: user, passwd = splitpasswd(user)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	515 else: passwd = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	516 host = unquote(host)
66269 f079643915b5 Merged revisions 86520 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 63134 diff changeset	517 user = user or ''
f079643915b5 Merged revisions 86520 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 63134 diff changeset	518 passwd = passwd or ''
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	519 host = socket.gethostbyname(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	520 if not port:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	521 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	522 port = ftplib.FTP_PORT
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	523 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	524 port = int(port)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	525 path, attrs = splitattr(path)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	526 path = unquote(path)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	527 dirs = path.split('/')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	528 dirs, file = dirs[:-1], dirs[-1]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	529 if dirs and not dirs[0]: dirs = dirs[1:]
11201 bf565b18cfdc Patches by Michael Reilly to correctly deal with ftp URLs of the form Guido van Rossum <guido@python.org> parents: 11094 diff changeset	530 if dirs and not dirs[0]: dirs[0] = '/'
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	531 key = user, host, port, '/'.join(dirs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	532 # XXX thread unsafe!
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	533 if len(self.ftpcache) > MAXFTPCACHE:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	534 # Prune the cache, rather arbitrarily
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	535 for k in self.ftpcache.keys():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	536 if k != key:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	537 v = self.ftpcache[k]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	538 del self.ftpcache[k]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	539 v.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	540 try:
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	541 if not key in self.ftpcache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	542 self.ftpcache[key] = \
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	543 ftpwrapper(user, passwd, host, port, dirs)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	544 if not file: type = 'D'
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	545 else: type = 'I'
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	546 for attr in attrs:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	547 attr, value = splitvalue(attr)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	548 if attr.lower() == 'type' and \
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	549 value in ('a', 'A', 'i', 'I', 'd', 'D'):
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	550 type = value.upper()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	551 (fp, retrlen) = self.ftpcache[key].retrfile(file, type)
19440 ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	552 mtype = mimetypes.guess_type("ftp:" + url)[0]
ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	553 headers = ""
ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	554 if mtype:
ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	555 headers += "Content-Type: %s\n" % mtype
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	556 if retrlen is not None and retrlen >= 0:
19440 ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	557 headers += "Content-Length: %d\n" % retrlen
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	558 headers = mimetools.Message(StringIO(headers))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	559 return addinfourl(fp, headers, "ftp:" + url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	560 except ftperrors(), msg:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	561 raise IOError, ('ftp error', msg), sys.exc_info()[2]
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	562
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	563 def open_data(self, url, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	564 """Use "data" URL."""
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	565 if not isinstance(url, str):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	566 raise IOError, ('data error', 'proxy support for data protocol currently not implemented')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	567 # ignore POSTed data
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	568 #
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	569 # syntax of data URLs:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	570 # dataurl := "data:" [ mediatype ] [ ";base64" ] "," data
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	571 # mediatype := [ type "/" subtype ] *( ";" parameter )
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	572 # data := *urlchar
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	573 # parameter := attribute "=" value
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	574 import mimetools
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	575 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	576 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	577 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	578 from StringIO import StringIO
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	579 try:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	580 [type, data] = url.split(',', 1)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	581 except ValueError:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	582 raise IOError, ('data error', 'bad data URL')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	583 if not type:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	584 type = 'text/plain;charset=US-ASCII'
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	585 semi = type.rfind(';')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	586 if semi >= 0 and '=' not in type[semi:]:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	587 encoding = type[semi+1:]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	588 type = type[:semi]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	589 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	590 encoding = ''
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	591 msg = []
61139 5b6a7f2a05c6 Merged revisions 80675 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60072 diff changeset	592 msg.append('Date: %s'%time.strftime('%a, %d %b %Y %H:%M:%S GMT',
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	593 time.gmtime(time.time())))
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	594 msg.append('Content-type: %s' % type)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	595 if encoding == 'base64':
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	596 data = base64.decodestring(data)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	597 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	598 data = unquote(data)
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	599 msg.append('Content-Length: %d' % len(data))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	600 msg.append('')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	601 msg.append(data)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	602 msg = '\n'.join(msg)
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	603 f = StringIO(msg)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	604 headers = mimetools.Message(f, 0)
35956 75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	605 #f.fileno = None # needed for addinfourl
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	606 return addinfourl(f, headers, url)
7666 c9394aeaac6a Added support for "data" URL, by Sjoerd Mullender. Guido van Rossum <guido@python.org> parents: 7272 diff changeset	607
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	608
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	609 class FancyURLopener(URLopener):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	610 """Derived class with handlers for errors we can handle (perhaps)."""
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	611
23781 730490ff25ea Fix SF #565414, FancyURLopener() needs to support *kwargs Neal Norwitz <nnorwitz@gmail.com>* parents: 23645 diff changeset	612 def __init__(self, args, *kwargs):
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	613 URLopener.__init__(self, args, *kwargs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	614 self.auth_cache = {}
16611 e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	615 self.tries = 0
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	616 self.maxtries = 10
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	617
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	618 def http_error_default(self, url, fp, errcode, errmsg, headers):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	619 """Default error handling -- don't raise an exception."""
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	620 return addinfourl(fp, headers, "http:" + url, errcode)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	621
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	622 def http_error_302(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	623 """Error 302 -- relocated (temporarily)."""
16611 e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	624 self.tries += 1
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	625 if self.maxtries and self.tries >= self.maxtries:
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	626 if hasattr(self, "http_error_500"):
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	627 meth = self.http_error_500
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	628 else:
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	629 meth = self.http_error_default
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	630 self.tries = 0
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	631 return meth(url, fp, 500,
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	632 "Internal Server Error: Redirect Recursion", headers)
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	633 result = self.redirect_internal(url, fp, errcode, errmsg, headers,
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	634 data)
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	635 self.tries = 0
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	636 return result
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	637
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	638 def redirect_internal(self, url, fp, errcode, errmsg, headers, data):
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	639 if 'location' in headers:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	640 newurl = headers['location']
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	641 elif 'uri' in headers:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	642 newurl = headers['uri']
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	643 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	644 return
69042 9d06d5eb1a7e Add tests for the urllib[2] vulnerability. Change to raise exceptions. guido@google.com parents: 69041 diff changeset	645 fp.close()
69040 dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	646 # In case the server sent a relative URL, join with original:
dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	647 newurl = basejoin(self.type + ":" + url, newurl)
dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	648
dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	649 # For security reasons we do not allow redirects to protocols
69041 ca3b117c40f3 Add FTP to the allowed url schemes. Add Misc/NEWS. guido@google.com parents: 69040 diff changeset	650 # other than HTTP, HTTPS or FTP.
69040 dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	651 newurl_lower = newurl.lower()
dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	652 if not (newurl_lower.startswith('http://') or
69041 ca3b117c40f3 Add FTP to the allowed url schemes. Add Misc/NEWS. guido@google.com parents: 69040 diff changeset	653 newurl_lower.startswith('https://') or
ca3b117c40f3 Add FTP to the allowed url schemes. Add Misc/NEWS. guido@google.com parents: 69040 diff changeset	654 newurl_lower.startswith('ftp://')):
69042 9d06d5eb1a7e Add tests for the urllib[2] vulnerability. Change to raise exceptions. guido@google.com parents: 69041 diff changeset	655 raise IOError('redirect error', errcode,
9d06d5eb1a7e Add tests for the urllib[2] vulnerability. Change to raise exceptions. guido@google.com parents: 69041 diff changeset	656 errmsg + " - Redirection to url '%s' is not allowed" %
9d06d5eb1a7e Add tests for the urllib[2] vulnerability. Change to raise exceptions. guido@google.com parents: 69041 diff changeset	657 newurl,
9d06d5eb1a7e Add tests for the urllib[2] vulnerability. Change to raise exceptions. guido@google.com parents: 69041 diff changeset	658 headers)
69040 dd852a0f92d6 Issue 22663: fix redirect vulnerability in urllib/urllib2. guido@google.com parents: 41187 diff changeset	659
29125 92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	660 return self.open(newurl)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	661
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	662 def http_error_301(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	663 """Error 301 -- also relocated (permanently)."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	664 return self.http_error_302(url, fp, errcode, errmsg, headers, data)
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	665
28771 29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	666 def http_error_303(self, url, fp, errcode, errmsg, headers, data=None):
29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	667 """Error 303 -- also relocated (essentially identical to 302)."""
29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	668 return self.http_error_302(url, fp, errcode, errmsg, headers, data)
29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	669
29125 92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	670 def http_error_307(self, url, fp, errcode, errmsg, headers, data=None):
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	671 """Error 307 -- relocated, but turn POST into error."""
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	672 if data is None:
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	673 return self.http_error_302(url, fp, errcode, errmsg, headers, data)
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	674 else:
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	675 return self.http_error_default(url, fp, errcode, errmsg, headers)
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	676
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	677 def http_error_401(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	678 """Error 401 -- authentication required.
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	679 This function supports Basic authentication only."""
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	680 if not 'www-authenticate' in headers:
16833 bbdeb6d4dea1 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16784 diff changeset	681 URLopener.http_error_default(self, url, fp,
20421 ee89567bcab4 Added missing parameter in call to http_error_default(); Fred Drake <fdrake@acm.org> parents: 19504 diff changeset	682 errcode, errmsg, headers)
16784 60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	683 stuff = headers['www-authenticate']
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	684 import re
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	685 match = re.match('[ \t]([^ \t]+)[ \t]+realm="([^"])"', stuff)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	686 if not match:
16833 bbdeb6d4dea1 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16784 diff changeset	687 URLopener.http_error_default(self, url, fp,
16784 60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	688 errcode, errmsg, headers)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	689 scheme, realm = match.groups()
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	690 if scheme.lower() != 'basic':
16833 bbdeb6d4dea1 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16784 diff changeset	691 URLopener.http_error_default(self, url, fp,
16784 60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	692 errcode, errmsg, headers)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	693 name = 'retry_' + self.type + '_basic_auth'
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	694 if data is None:
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	695 return getattr(self,name)(url, realm)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	696 else:
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	697 return getattr(self,name)(url, realm, data)
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	698
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	699 def http_error_407(self, url, fp, errcode, errmsg, headers, data=None):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	700 """Error 407 -- proxy authentication required.
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	701 This function supports Basic authentication only."""
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	702 if not 'proxy-authenticate' in headers:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	703 URLopener.http_error_default(self, url, fp,
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	704 errcode, errmsg, headers)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	705 stuff = headers['proxy-authenticate']
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	706 import re
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	707 match = re.match('[ \t]([^ \t]+)[ \t]+realm="([^"])"', stuff)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	708 if not match:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	709 URLopener.http_error_default(self, url, fp,
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	710 errcode, errmsg, headers)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	711 scheme, realm = match.groups()
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	712 if scheme.lower() != 'basic':
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	713 URLopener.http_error_default(self, url, fp,
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	714 errcode, errmsg, headers)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	715 name = 'retry_proxy_' + self.type + '_basic_auth'
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	716 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	717 return getattr(self,name)(url, realm)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	718 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	719 return getattr(self,name)(url, realm, data)
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	720
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	721 def retry_proxy_http_basic_auth(self, url, realm, data=None):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	722 host, selector = splithost(url)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	723 newurl = 'http://' + host + selector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	724 proxy = self.proxies['http']
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	725 urltype, proxyhost = splittype(proxy)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	726 proxyhost, proxyselector = splithost(proxyhost)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	727 i = proxyhost.find('@') + 1
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	728 proxyhost = proxyhost[i:]
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	729 user, passwd = self.get_user_passwd(proxyhost, realm, i)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	730 if not (user or passwd): return None
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	731 proxyhost = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + proxyhost
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	732 self.proxies['http'] = 'http://' + proxyhost + proxyselector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	733 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	734 return self.open(newurl)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	735 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	736 return self.open(newurl, data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	737
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	738 def retry_proxy_https_basic_auth(self, url, realm, data=None):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	739 host, selector = splithost(url)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	740 newurl = 'https://' + host + selector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	741 proxy = self.proxies['https']
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	742 urltype, proxyhost = splittype(proxy)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	743 proxyhost, proxyselector = splithost(proxyhost)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	744 i = proxyhost.find('@') + 1
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	745 proxyhost = proxyhost[i:]
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	746 user, passwd = self.get_user_passwd(proxyhost, realm, i)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	747 if not (user or passwd): return None
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	748 proxyhost = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + proxyhost
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	749 self.proxies['https'] = 'https://' + proxyhost + proxyselector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	750 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	751 return self.open(newurl)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	752 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	753 return self.open(newurl, data)
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	754
11688 1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	755 def retry_http_basic_auth(self, url, realm, data=None):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	756 host, selector = splithost(url)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	757 i = host.find('@') + 1
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	758 host = host[i:]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	759 user, passwd = self.get_user_passwd(host, realm, i)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	760 if not (user or passwd): return None
15889 b0ba46b16e07 - Make sure to quote the username and password (SF patch #103236 by Guido van Rossum <guido@python.org> parents: 15874 diff changeset	761 host = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + host
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	762 newurl = 'http://' + host + selector
11688 1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	763 if data is None:
1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	764 return self.open(newurl)
1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	765 else:
1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	766 return self.open(newurl, data)
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	767
11688 1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	768 def retry_https_basic_auth(self, url, realm, data=None):
15874 4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	769 host, selector = splithost(url)
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	770 i = host.find('@') + 1
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	771 host = host[i:]
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	772 user, passwd = self.get_user_passwd(host, realm, i)
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	773 if not (user or passwd): return None
15889 b0ba46b16e07 - Make sure to quote the username and password (SF patch #103236 by Guido van Rossum <guido@python.org> parents: 15874 diff changeset	774 host = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + host
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	775 newurl = 'https://' + host + selector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	776 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	777 return self.open(newurl)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	778 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	779 return self.open(newurl, data)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	780
61661 0b04c5ca577d Slight style cleanup. Florent Xicluna <florent.xicluna@gmail.com> parents: 61258 diff changeset	781 def get_user_passwd(self, host, realm, clear_cache=0):
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	782 key = realm + '@' + host.lower()
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	783 if key in self.auth_cache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	784 if clear_cache:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	785 del self.auth_cache[key]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	786 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	787 return self.auth_cache[key]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	788 user, passwd = self.prompt_user_passwd(host, realm)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	789 if user or passwd: self.auth_cache[key] = (user, passwd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	790 return user, passwd
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	791
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	792 def prompt_user_passwd(self, host, realm):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	793 """Override this in a GUI environment!"""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	794 import getpass
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	795 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	796 user = raw_input("Enter username for %s at %s: " % (realm,
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	797 host))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	798 passwd = getpass.getpass("Enter password for %s in %s at %s: " %
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	799 (user, realm, host))
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	800 return user, passwd
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	801 except KeyboardInterrupt:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	802 print
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	803 return None, None
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	804
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	805
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	806 # Utility functions
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	807
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	808 _localhost = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	809 def localhost():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	810 """Return the IP address of the magic hostname 'localhost'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	811 global _localhost
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	812 if _localhost is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	813 _localhost = socket.gethostbyname('localhost')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	814 return _localhost
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	815
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	816 _thishost = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	817 def thishost():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	818 """Return the IP address of the current host."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	819 global _thishost
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	820 if _thishost is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	821 _thishost = socket.gethostbyname(socket.gethostname())
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	822 return _thishost
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	823
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	824 _ftperrors = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	825 def ftperrors():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	826 """Return the set of errors raised by the FTP class."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	827 global _ftperrors
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	828 if _ftperrors is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	829 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	830 _ftperrors = ftplib.all_errors
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	831 return _ftperrors
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	832
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	833 _noheaders = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	834 def noheaders():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	835 """Return an empty mimetools.Message object."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	836 global _noheaders
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	837 if _noheaders is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	838 import mimetools
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	839 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	840 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	841 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	842 from StringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	843 _noheaders = mimetools.Message(StringIO(), 0)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	844 _noheaders.fp.close() # Recycle file descriptor
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	845 return _noheaders
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	846
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	847
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	848 # Utility classes
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	849
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	850 class ftpwrapper:
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	851 """Class used by open_ftp() for cache of open FTP connections."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	852
47372 e2c5ae30cffe Fixed the semantic of timeout for socket.create_connection and Facundo Batista <facundobatista@gmail.com> parents: 47162 diff changeset	853 def __init__(self, user, passwd, host, port, dirs,
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	854 timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
71480 dbacafae72d8 Fix typo in dbf1e1a27427 that was causing some buildbots to fail. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 71478 diff changeset	855 persistent=True):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	856 self.user = user
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	857 self.passwd = passwd
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	858 self.host = host
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	859 self.port = port
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	860 self.dirs = dirs
41854 10426ad5a6e0 Added an optional timeout parameter to urllib.ftpwrapper, with tests Facundo Batista <facundobatista@gmail.com> parents: 41743 diff changeset	861 self.timeout = timeout
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	862 self.refcount = 0
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	863 self.keepalive = persistent
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	864 self.init()
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	865
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	866 def init(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	867 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	868 self.busy = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	869 self.ftp = ftplib.FTP()
41854 10426ad5a6e0 Added an optional timeout parameter to urllib.ftpwrapper, with tests Facundo Batista <facundobatista@gmail.com> parents: 41743 diff changeset	870 self.ftp.connect(self.host, self.port, self.timeout)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	871 self.ftp.login(self.user, self.passwd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	872 for dir in self.dirs:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	873 self.ftp.cwd(dir)
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	874
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	875 def retrfile(self, file, type):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	876 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	877 self.endtransfer()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	878 if type in ('d', 'D'): cmd = 'TYPE A'; isdir = 1
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	879 else: cmd = 'TYPE ' + type; isdir = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	880 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	881 self.ftp.voidcmd(cmd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	882 except ftplib.all_errors:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	883 self.init()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	884 self.ftp.voidcmd(cmd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	885 conn = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	886 if file and not isdir:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	887 # Try to retrieve as a file
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	888 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	889 cmd = 'RETR ' + file
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	890 conn, retrlen = self.ftp.ntransfercmd(cmd)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	891 except ftplib.error_perm, reason:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	892 if str(reason)[:3] != '550':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	893 raise IOError, ('ftp error', reason), sys.exc_info()[2]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	894 if not conn:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	895 # Set transfer mode to ASCII!
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	896 self.ftp.voidcmd('TYPE A')
44819 abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	897 # Try a directory listing. Verify that directory exists.
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	898 if file:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	899 pwd = self.ftp.pwd()
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	900 try:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	901 try:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	902 self.ftp.cwd(file)
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	903 except ftplib.error_perm, reason:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	904 raise IOError, ('ftp error', reason), sys.exc_info()[2]
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	905 finally:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	906 self.ftp.cwd(pwd)
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	907 cmd = 'LIST ' + file
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	908 else:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	909 cmd = 'LIST'
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	910 conn, retrlen = self.ftp.ntransfercmd(cmd)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	911 self.busy = 1
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	912 ftpobj = addclosehook(conn.makefile('rb'), self.file_close)
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	913 self.refcount += 1
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	914 conn.close()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	915 # Pass back both a suitably decorated object and a retrieval length
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	916 return (ftpobj, retrlen)
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	917
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	918 def endtransfer(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	919 if not self.busy:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	920 return
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	921 self.busy = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	922 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	923 self.ftp.voidresp()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	924 except ftperrors():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	925 pass
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	926
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	927 def close(self):
71478 dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	928 self.keepalive = False
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	929 if self.refcount <= 0:
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	930 self.real_close()
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	931
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	932 def file_close(self):
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	933 self.endtransfer()
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	934 self.refcount -= 1
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	935 if self.refcount <= 0 and not self.keepalive:
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	936 self.real_close()
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	937
dbf1e1a27427 Issue #10883: Fix socket leaks in urllib.request. Nadeem Vawda <nadeem.vawda@gmail.com> parents: 69046 diff changeset	938 def real_close(self):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	939 self.endtransfer()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	940 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	941 self.ftp.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	942 except ftperrors():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	943 pass
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	944
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	945 class addbase:
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	946 """Base class for addinfo and addclosehook."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	947
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	948 def __init__(self, fp):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	949 self.fp = fp
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	950 self.read = self.fp.read
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	951 self.readline = self.fp.readline
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	952 if hasattr(self.fp, "readlines"): self.readlines = self.fp.readlines
35956 75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	953 if hasattr(self.fp, "fileno"):
75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	954 self.fileno = self.fp.fileno
75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	955 else:
75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	956 self.fileno = lambda: None
28239 8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	957 if hasattr(self.fp, "__iter__"):
8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	958 self.__iter__ = self.fp.__iter__
8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	959 if hasattr(self.fp, "next"):
8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	960 self.next = self.fp.next
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	961
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	962 def __repr__(self):
31515 71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	963 return '<%s at %r whose fp = %r>' % (self.__class__.__name__,
71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	964 id(self), self.fp)
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	965
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	966 def close(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	967 self.read = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	968 self.readline = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	969 self.readlines = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	970 self.fileno = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	971 if self.fp: self.fp.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	972 self.fp = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	973
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	974 class addclosehook(addbase):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	975 """Class to add a close hook to an open file."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	976
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	977 def __init__(self, fp, closehook, *hookargs):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	978 addbase.__init__(self, fp)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	979 self.closehook = closehook
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	980 self.hookargs = hookargs
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	981
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	982 def close(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	983 if self.closehook:
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	984 self.closehook(*self.hookargs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	985 self.closehook = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	986 self.hookargs = None
75710 6ce4868861ba Fix the urllib closing issue which hangs on particular ftp urls/ftp servers. closes issue11199 Senthil Kumaran <senthil@uthcode.com> parents: 74546 diff changeset	987 addbase.close(self)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	988
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	989 class addinfo(addbase):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	990 """class to add an info() method to an open file."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	991
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	992 def __init__(self, fp, headers):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	993 addbase.__init__(self, fp)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	994 self.headers = headers
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	995
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	996 def info(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	997 return self.headers
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	998
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	999 class addinfourl(addbase):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1000 """class to add info() and geturl() methods to an open file."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1001
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	1002 def __init__(self, fp, headers, url, code=None):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1003 addbase.__init__(self, fp)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1004 self.headers = headers
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1005 self.url = url
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	1006 self.code = code
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1007
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1008 def info(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1009 return self.headers
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1010
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	1011 def getcode(self):
3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	1012 return self.code
3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	1013
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1014 def geturl(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1015 return self.url
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	1016
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1017
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1018 # Utilities to parse URLs (most of these return None for missing parts):
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	1019 # unwrap('<URL:type://host/path>') --> 'type://host/path'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1020 # splittype('type:opaquestring') --> 'type', 'opaquestring'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1021 # splithost('//host[:port]/path') --> 'host[:port]', '/path'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1022 # splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1023 # splitpasswd('user:passwd') -> 'user', 'passwd'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1024 # splitport('host:port') --> 'host', 'port'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1025 # splitquery('/path?query') --> '/path', 'query'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1026 # splittag('/path#tag') --> '/path', 'tag'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1027 # splitattr('/path;attr1=value1;attr2=value2;...') ->
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1028 # '/path', ['attr1=value1', 'attr2=value2', ...]
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1029 # splitvalue('attr=value') --> 'attr', 'value'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1030 # unquote('abc%20def') -> 'abc def'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1031 # quote('abc def') -> 'abc%20def')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1032
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1033 try:
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1034 unicode
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1035 except NameError:
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1036 def _is_unicode(x):
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1037 return 0
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1038 else:
23540 bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1039 def _is_unicode(x):
bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1040 return isinstance(x, unicode)
bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1041
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1042 def toBytes(url):
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1043 """toBytes(u"URL") --> 'URL'."""
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1044 # Most URL schemes require ASCII. If that changes, the conversion
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1045 # can be relaxed
23540 bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1046 if _is_unicode(url):
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1047 try:
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1048 url = url.encode("ASCII")
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1049 except UnicodeError:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1050 raise UnicodeError("URL " + repr(url) +
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1051 " contains non-ASCII characters")
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1052 return url
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1053
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1054 def unwrap(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1055 """unwrap('<URL:type://host/path>') --> 'type://host/path'."""
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1056 url = url.strip()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1057 if url[:1] == '<' and url[-1:] == '>':
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1058 url = url[1:-1].strip()
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1059 if url[:4] == 'URL:': url = url[4:].strip()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1060 return url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1061
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1062 _typeprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1063 def splittype(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1064 """splittype('type:opaquestring') --> 'type', 'opaquestring'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1065 global _typeprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1066 if _typeprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1067 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1068 _typeprog = re.compile('^([^/:]+):')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1069
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1070 match = _typeprog.match(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1071 if match:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1072 scheme = match.group(1)
13287 4011acfe1433 Fix bug #314, reported by Craig Allen <cba@mediaone.net>: Fred Drake <fdrake@acm.org> parents: 12905 diff changeset	1073 return scheme.lower(), url[len(scheme) + 1:]
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1074 return None, url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1075
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1076 _hostprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1077 def splithost(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1078 """splithost('//host[:port]/path') --> 'host[:port]', '/path'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1079 global _hostprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1080 if _hostprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1081 import re
37228 8f7c03acaddd Bug #1457264: parse http://host?query correctly in urllib Georg Brandl <georg@python.org> parents: 36576 diff changeset	1082 _hostprog = re.compile('^//([^/?])(.)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1083
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1084 match = _hostprog.match(url)
66389 4c2b2be71457 Merged revisions 86676 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 66269 diff changeset	1085 if match:
4c2b2be71457 Merged revisions 86676 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 66269 diff changeset	1086 host_port = match.group(1)
4c2b2be71457 Merged revisions 86676 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 66269 diff changeset	1087 path = match.group(2)
4c2b2be71457 Merged revisions 86676 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 66269 diff changeset	1088 if path and not path.startswith('/'):
4c2b2be71457 Merged revisions 86676 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 66269 diff changeset	1089 path = '/' + path
4c2b2be71457 Merged revisions 86676 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 66269 diff changeset	1090 return host_port, path
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1091 return None, url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1092
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1093 _userprog = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1094 def splituser(host):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1095 """splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1096 global _userprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1097 if _userprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1098 import re
24880 710bebe2182c Modify splituser() method to allow an @ in the userinfo field. Raymond Hettinger <python@rcn.com> parents: 24710 diff changeset	1099 _userprog = re.compile('^(.)@(.)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1100
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1101 match = _userprog.match(host)
66269 f079643915b5 Merged revisions 86520 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 63134 diff changeset	1102 if match: return match.group(1, 2)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1103 return None, host
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1104
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1105 _passwdprog = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1106 def splitpasswd(user):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1107 """splitpasswd('user:passwd') -> 'user', 'passwd'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1108 global _passwdprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1109 if _passwdprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1110 import re
52476 2ff89f0f1d19 Fix for bugs: Issue4675 and Issue4962. Senthil Kumaran <orsenthil@gmail.com> parents: 52265 diff changeset	1111 _passwdprog = re.compile('^([^:]):(.)$',re.S)
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1112
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1113 match = _passwdprog.match(user)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1114 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1115 return user, None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1116
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1117 # splittag('/path#tag') --> '/path', 'tag'
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1118 _portprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1119 def splitport(host):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1120 """splitport('host:port') --> 'host', 'port'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1121 global _portprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1122 if _portprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1123 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1124 _portprog = re.compile('^(.*):([0-9]+)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1125
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1126 match = _portprog.match(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1127 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1128 return host, None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1129
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1130 _nportprog = None
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	1131 def splitnport(host, defport=-1):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1132 """Split host and port, returning numeric port.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1133 Return given default port if no ':' found; defaults to -1.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1134 Return numerical port if a valid number are found after ':'.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1135 Return None if ':' but not a valid number."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1136 global _nportprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1137 if _nportprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1138 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1139 _nportprog = re.compile('^(.):(.)$')
7753 8ef067559d5a A few lines were indented using spaces instead of tabs -- fix them. Guido van Rossum <guido@python.org> parents: 7666 diff changeset	1140
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1141 match = _nportprog.match(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1142 if match:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1143 host, port = match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1144 try:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1145 if not port: raise ValueError, "no digits"
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1146 nport = int(port)
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1147 except ValueError:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1148 nport = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1149 return host, nport
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1150 return host, defport
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	1151
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1152 _queryprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1153 def splitquery(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1154 """splitquery('/path?query') --> '/path', 'query'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1155 global _queryprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1156 if _queryprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1157 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1158 _queryprog = re.compile('^(.)\?([^?])$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1159
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1160 match = _queryprog.match(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1161 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1162 return url, None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1163
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1164 _tagprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1165 def splittag(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1166 """splittag('/path#tag') --> '/path', 'tag'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1167 global _tagprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1168 if _tagprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1169 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1170 _tagprog = re.compile('^(.)#([^#])$')
7753 8ef067559d5a A few lines were indented using spaces instead of tabs -- fix them. Guido van Rossum <guido@python.org> parents: 7666 diff changeset	1171
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1172 match = _tagprog.match(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1173 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1174 return url, None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1175
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1176 def splitattr(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1177 """splitattr('/path;attr1=value1;attr2=value2;...') ->
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1178 '/path', ['attr1=value1', 'attr2=value2', ...]."""
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1179 words = url.split(';')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1180 return words[0], words[1:]
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1181
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1182 _valueprog = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1183 def splitvalue(attr):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1184 """splitvalue('attr=value') --> 'attr', 'value'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1185 global _valueprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1186 if _valueprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1187 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1188 _valueprog = re.compile('^([^=])=(.)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1189
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1190 match = _valueprog.match(attr)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1191 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1192 return attr, None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1193
61901 2297cd57d945 Issue 8143: sync unquote in urlparse with urllib; add comment about doing so. R. David Murray <rdmurray@bitdance.com> parents: 61667 diff changeset	1194 # urlparse contains a duplicate of this method to avoid a circular import. If
2297cd57d945 Issue 8143: sync unquote in urlparse with urllib; add comment about doing so. R. David Murray <rdmurray@bitdance.com> parents: 61667 diff changeset	1195 # you update this method, also update the copy in urlparse. This code
2297cd57d945 Issue 8143: sync unquote in urlparse with urllib; add comment about doing so. R. David Murray <rdmurray@bitdance.com> parents: 61667 diff changeset	1196 # duplication does not exist in Python3.
2297cd57d945 Issue 8143: sync unquote in urlparse with urllib; add comment about doing so. R. David Murray <rdmurray@bitdance.com> parents: 61667 diff changeset	1197
60072 c3c82d8040d6 Merged revisions 79047 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 58968 diff changeset	1198 _hexdig = '0123456789ABCDEFabcdef'
61661 0b04c5ca577d Slight style cleanup. Florent Xicluna <florent.xicluna@gmail.com> parents: 61258 diff changeset	1199 _hextochr = dict((a + b, chr(int(a + b, 16)))
0b04c5ca577d Slight style cleanup. Florent Xicluna <florent.xicluna@gmail.com> parents: 61258 diff changeset	1200 for a in _hexdig for b in _hexdig)
35529 dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1201
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1202 def unquote(s):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1203 """unquote('abc%20def') -> 'abc def'."""
35529 dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1204 res = s.split('%')
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1205 # fastpath
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1206 if len(res) == 1:
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1207 return s
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1208 s = res[0]
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1209 for item in res[1:]:
35529 dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1210 try:
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1211 s += _hextochr[item[:2]] + item[2:]
35529 dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1212 except KeyError:
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1213 s += '%' + item
35723 ae539af4afc5 Teach unquote() to handle unicode inputs Raymond Hettinger <python@rcn.com> parents: 35531 diff changeset	1214 except UnicodeDecodeError:
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1215 s += unichr(int(item[:2], 16)) + item[2:]
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1216 return s
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1217
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	1218 def unquote_plus(s):
14118 44f4a84a80a8 * added doc strings to urlopen and unquote_plus Skip Montanaro <skip@pobox.com> parents: 14108 diff changeset	1219 """unquote('%7e/abc+def') -> '~/abc def'"""
31794 f1d44295ebf7 Replace sequential split/join calls on strings with a single replace call. Brett Cannon <bcannon@gmail.com> parents: 31788 diff changeset	1220 s = s.replace('+', ' ')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1221 return unquote(s)
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	1222
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1223 always_safe = ('ABCDEFGHIJKLMNOPQRSTUVWXYZ'
14297 eda449f54e44 fixes bug #111951 Jeremy Hylton <jeremy@alum.mit.edu> parents: 14181 diff changeset	1224 'abcdefghijklmnopqrstuvwxyz'
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1225 '0123456789' '_.-')
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1226 _safe_map = {}
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1227 for i, c in zip(xrange(256), str(bytearray(xrange(256)))):
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1228 _safe_map[c] = c if (i < 128 and c in always_safe) else '%{:02X}'.format(i)
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1229 _safe_quoters = {}
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1230
63134 1f4ed71b68e2 Reverting the checkin made in revision 82940, as it was adding new parameters to quote function in a bugfix release. Senthil Kumaran <orsenthil@gmail.com> parents: 63099 diff changeset	1231 def quote(s, safe='/'):
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1232 """quote('abc def') -> 'abc%20def'
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1233
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1234 Each part of a URL, e.g. the path info, the query, etc., has a
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1235 different set of reserved characters that must be quoted.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1236
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1237 RFC 2396 Uniform Resource Identifiers (URI): Generic Syntax lists
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1238 the following reserved characters.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1239
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1240 reserved = ";" \| "/" \| "?" \| ":" \| "@" \| "&" \| "=" \| "+" \|
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1241 "$" \| ","
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1242
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1243 Each of these characters is reserved in some component of a URL,
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1244 but not necessarily in all of them.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1245
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1246 By default, the quote function is intended for quoting the path
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1247 section of a URL. Thus, it will not encode '/'. This character
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1248 is reserved, but in typical usage the quote function is being
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1249 called on a path where the existing slash characters are used as
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1250 reserved characters.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1251 """
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1252 # fastpath
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1253 if not s:
63099 a1121409f96b Fix Issue9301 - urllib.quote(None) to raise TypeError Senthil Kumaran <orsenthil@gmail.com> parents: 63064 diff changeset	1254 if s is None:
a1121409f96b Fix Issue9301 - urllib.quote(None) to raise TypeError Senthil Kumaran <orsenthil@gmail.com> parents: 63064 diff changeset	1255 raise TypeError('None object cannot be quoted')
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1256 return s
35527 5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1257 cachekey = (safe, always_safe)
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1258 try:
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1259 (quoter, safe) = _safe_quoters[cachekey]
35527 5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1260 except KeyError:
61667 c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1261 safe_map = _safe_map.copy()
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1262 safe_map.update([(c, c) for c in safe])
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1263 quoter = safe_map.__getitem__
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1264 safe = always_safe + safe
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1265 _safe_quoters[cachekey] = (quoter, safe)
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1266 if not s.rstrip(safe):
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1267 return s
c981f0c8678b Issue #1285086: Speed up urllib.quote and urllib.unquote for simple cases. Florent Xicluna <florent.xicluna@gmail.com> parents: 61661 diff changeset	1268 return ''.join(map(quoter, s))
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1269
63134 1f4ed71b68e2 Reverting the checkin made in revision 82940, as it was adding new parameters to quote function in a bugfix release. Senthil Kumaran <orsenthil@gmail.com> parents: 63099 diff changeset	1270 def quote_plus(s, safe=''):
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1271 """Quote the query fragment of a URL; replacing ' ' with '+'"""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1272 if ' ' in s:
35531 39dbe3be327a Corrected version of 1.170 Raymond Hettinger <python@rcn.com> parents: 35530 diff changeset	1273 s = quote(s, safe + ' ')
39dbe3be327a Corrected version of 1.170 Raymond Hettinger <python@rcn.com> parents: 35530 diff changeset	1274 return s.replace(' ', '+')
39dbe3be327a Corrected version of 1.170 Raymond Hettinger <python@rcn.com> parents: 35530 diff changeset	1275 return quote(s, safe)
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	1276
61661 0b04c5ca577d Slight style cleanup. Florent Xicluna <florent.xicluna@gmail.com> parents: 61258 diff changeset	1277 def urlencode(query, doseq=0):
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1278 """Encode a sequence of two-element tuples or dictionary into a URL query string.
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1279
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1280 If any values in the query arg are sequences and doseq is true, each
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1281 sequence element is converted to a separate parameter.
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1282
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1283 If the query arg is a sequence of two-element tuples, the order of the
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1284 parameters in the output will match the order of parameters in the
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1285 input.
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1286 """
16535 084b3dc9b6ff Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16324 diff changeset	1287
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1288 if hasattr(query,"items"):
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1289 # mapping objects
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1290 query = query.items()
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1291 else:
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1292 # it's a bother at times that strings and string-like objects are
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1293 # sequences...
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1294 try:
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1295 # non-sequence items should not work with len()
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1296 # non-empty strings will fail this
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1297 if len(query) and not isinstance(query[0], tuple):
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1298 raise TypeError
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1299 # zero-length sequences of all types will get here and succeed,
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1300 # but that's a minor nit - since the original implementation
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1301 # allowed empty dicts that type of behavior probably should be
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1302 # preserved for consistency
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1303 except TypeError:
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1304 ty,va,tb = sys.exc_info()
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1305 raise TypeError, "not a valid non-string sequence or mapping object", tb
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1306
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1307 l = []
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1308 if not doseq:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1309 # preserve old behavior
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1310 for k, v in query:
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1311 k = quote_plus(str(k))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1312 v = quote_plus(str(v))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1313 l.append(k + '=' + v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1314 else:
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1315 for k, v in query:
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1316 k = quote_plus(str(k))
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1317 if isinstance(v, str):
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1318 v = quote_plus(v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1319 l.append(k + '=' + v)
23540 bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1320 elif _is_unicode(v):
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1321 # is there a reasonable way to convert to ASCII?
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1322 # encode generates a string, but "replace" or "ignore"
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1323 # lose information and "strict" can raise UnicodeError
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1324 v = quote_plus(v.encode("ASCII","replace"))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1325 l.append(k + '=' + v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1326 else:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1327 try:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1328 # is this a sufficient test for sequence-ness?
58761 cc367c077245 No need to assign the results of expressions used only for side effects. Georg Brandl <georg@python.org> parents: 58759 diff changeset	1329 len(v)
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1330 except TypeError:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1331 # not a sequence
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1332 v = quote_plus(str(v))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1333 l.append(k + '=' + v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1334 else:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1335 # loop over the sequence
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1336 for elt in v:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1337 l.append(k + '=' + quote_plus(str(elt)))
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1338 return '&'.join(l)
8700 95f324272f54 Speed up the implementation of quote(). Guido van Rossum <guido@python.org> parents: 8579 diff changeset	1339
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1340 # Proxy handling
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1341 def getproxies_environment():
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1342 """Return a dictionary of scheme -> proxy server URL mappings.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1343
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1344 Scan the environment for variables named <scheme>_proxy;
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1345 this seems to be the standard convention. If you need a
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1346 different way, you can pass a proxies dictionary to the
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1347 [Fancy]URLopener constructor.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1348
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1349 """
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1350 proxies = {}
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1351 for name, value in os.environ.items():
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1352 name = name.lower()
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1353 if value and name[-6:] == '_proxy':
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1354 proxies[name[:-6]] = value
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1355 return proxies
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1356
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1357 def proxy_bypass_environment(host):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1358 """Test if proxies should not be used for a particular host.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1359
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1360 Checks the environment for a variable named no_proxy, which should
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1361 be a list of DNS suffixes separated by commas, or '*' for all hosts.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1362 """
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1363 no_proxy = os.environ.get('no_proxy', '') or os.environ.get('NO_PROXY', '')
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1364 # '*' is special case for always bypass
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1365 if no_proxy == '*':
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1366 return 1
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1367 # strip port off host
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1368 hostonly, port = splitport(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1369 # check if the host ends with any of the DNS suffixes
71750 c5a35bcfa3ee Fix closes issue12698 - make the no_proxy environment variable handling a bit lenient (accomodate spaces in between the items) Senthil Kumaran <senthil@uthcode.com> parents: 71480 diff changeset	1370 no_proxy_list = [proxy.strip() for proxy in no_proxy.split(',')]
c5a35bcfa3ee Fix closes issue12698 - make the no_proxy environment variable handling a bit lenient (accomodate spaces in between the items) Senthil Kumaran <senthil@uthcode.com> parents: 71480 diff changeset	1371 for name in no_proxy_list:
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1372 if name and (hostonly.endswith(name) or host.endswith(name)):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1373 return 1
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1374 # otherwise, don't bypass
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1375 return 0
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1376
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1377
32710 df88fadba2f5 Fix for #779167: use InternetConfig proxy settings on MacOSX (in addition Jack Jansen <jack.jansen@cwi.nl> parents: 31794 diff changeset	1378 if sys.platform == 'darwin':
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1379 from _scproxy import _get_proxy_settings, _get_proxies
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1380
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1381 def proxy_bypass_macosx_sysconf(host):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1382 """
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1383 Return True iff this host shouldn't be accessed using a proxy
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1384
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1385 This function uses the MacOSX framework SystemConfiguration
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1386 to fetch the proxy information.
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1387 """
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1388 import re
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1389 import socket
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1390 from fnmatch import fnmatch
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1391
56582 ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1392 hostonly, port = splitport(host)
ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1393
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1394 def ip2num(ipAddr):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1395 parts = ipAddr.split('.')
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1396 parts = map(int, parts)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1397 if len(parts) != 4:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1398 parts = (parts + [0, 0, 0, 0])[:4]
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1399 return (parts[0] << 24) \| (parts[1] << 16) \| (parts[2] << 8) \| parts[3]
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1400
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1401 proxy_settings = _get_proxy_settings()
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1402
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1403 # Check for simple host names:
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1404 if '.' not in host:
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1405 if proxy_settings['exclude_simple']:
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1406 return True
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1407
56582 ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1408 hostIP = None
ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1409
56129 d131956533bf Merged revisions 74964 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56127 diff changeset	1410 for value in proxy_settings.get('exceptions', ()):
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1411 # Items in the list are strings like these: *.local, 169.254/16
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1412 if not value: continue
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1413
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1414 m = re.match(r"(\d+(?:\.\d+)*)(/\d+)?", value)
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1415 if m is not None:
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1416 if hostIP is None:
56582 ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1417 try:
ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1418 hostIP = socket.gethostbyname(hostonly)
ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1419 hostIP = ip2num(hostIP)
ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1420 except socket.error:
ed7dec136430 Merged revisions 75482 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 56129 diff changeset	1421 continue
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1422
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1423 base = ip2num(m.group(1))
62553 deed156d33d8 Merged revisions 82284 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 61139 diff changeset	1424 mask = m.group(2)
deed156d33d8 Merged revisions 82284 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 61139 diff changeset	1425 if mask is None:
deed156d33d8 Merged revisions 82284 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 61139 diff changeset	1426 mask = 8 * (m.group(1).count('.') + 1)
deed156d33d8 Merged revisions 82284 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 61139 diff changeset	1427
deed156d33d8 Merged revisions 82284 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 61139 diff changeset	1428 else:
deed156d33d8 Merged revisions 82284 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 61139 diff changeset	1429 mask = int(mask[1:])
68476 8f0756010720 Issue #11500: Fixed a bug in the os x proxy bypass code for fully qualified Ronald Oussoren <ronaldoussoren@mac.com> parents: 68256 diff changeset	1430 mask = 32 - mask
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1431
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1432 if (hostIP >> mask) == (base >> mask):
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1433 return True
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1434
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1435 elif fnmatch(host, value):
2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1436 return True
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1437
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1438 return False
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1439
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1440 def getproxies_macosx_sysconf():
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1441 """Return a dictionary of scheme -> proxy server URL mappings.
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1442
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1443 This function uses the MacOSX framework SystemConfiguration
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1444 to fetch the proxy information.
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1445 """
56127 2717d85aba2e Merged revisions 74962 via svnmerge from Ronald Oussoren <ronaldoussoren@mac.com> parents: 55649 diff changeset	1446 return _get_proxies()
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1447
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1448 def proxy_bypass(host):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1449 if getproxies_environment():
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1450 return proxy_bypass_environment(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1451 else:
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1452 return proxy_bypass_macosx_sysconf(host)
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1453
32710 df88fadba2f5 Fix for #779167: use InternetConfig proxy settings on MacOSX (in addition Jack Jansen <jack.jansen@cwi.nl> parents: 31794 diff changeset	1454 def getproxies():
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1455 return getproxies_environment() or getproxies_macosx_sysconf()
32736 c3a096bfbd6b Whitespace normalization, via reindent.py. Tim Peters <tim.peters@gmail.com> parents: 32710 diff changeset	1456
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1457 elif os.name == 'nt':
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1458 def getproxies_registry():
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1459 """Return a dictionary of scheme -> proxy server URL mappings.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1460
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1461 Win32 uses the registry to store proxies.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1462
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1463 """
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1464 proxies = {}
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1465 try:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1466 import _winreg
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1467 except ImportError:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1468 # Std module, so should be around - but you never know!
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1469 return proxies
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1470 try:
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1471 internetSettings = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER,
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1472 r'Software\Microsoft\Windows\CurrentVersion\Internet Settings')
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1473 proxyEnable = _winreg.QueryValueEx(internetSettings,
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1474 'ProxyEnable')[0]
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1475 if proxyEnable:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1476 # Returned as Unicode but problems if not converted to ASCII
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1477 proxyServer = str(_winreg.QueryValueEx(internetSettings,
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1478 'ProxyServer')[0])
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1479 if '=' in proxyServer:
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1480 # Per-protocol settings
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1481 for p in proxyServer.split(';'):
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1482 protocol, address = p.split('=', 1)
22717 0bdd413a3416 Hopeful fix for SF bug 503031: urllib.py: open_http() host problem. Guido van Rossum <guido@python.org> parents: 22588 diff changeset	1483 # See if address has a type:// prefix
22768 3a5065005fcf Fix for a bug in the fix for SF bug 503031. This time the OP verified Guido van Rossum <guido@python.org> parents: 22717 diff changeset	1484 import re
3a5065005fcf Fix for a bug in the fix for SF bug 503031. This time the OP verified Guido van Rossum <guido@python.org> parents: 22717 diff changeset	1485 if not re.match('^([^/:]+)://', address):
22717 0bdd413a3416 Hopeful fix for SF bug 503031: urllib.py: open_http() host problem. Guido van Rossum <guido@python.org> parents: 22588 diff changeset	1486 address = '%s://%s' % (protocol, address)
0bdd413a3416 Hopeful fix for SF bug 503031: urllib.py: open_http() host problem. Guido van Rossum <guido@python.org> parents: 22588 diff changeset	1487 proxies[protocol] = address
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1488 else:
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1489 # Use one setting for all protocols
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1490 if proxyServer[:5] == 'http:':
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1491 proxies['http'] = proxyServer
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1492 else:
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1493 proxies['http'] = 'http://%s' % proxyServer
63035 6333d3a7e5c7 Merged revisions 82897 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 63031 diff changeset	1494 proxies['https'] = 'https://%s' % proxyServer
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1495 proxies['ftp'] = 'ftp://%s' % proxyServer
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1496 internetSettings.Close()
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1497 except (WindowsError, ValueError, TypeError):
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1498 # Either registry key not found etc, or the value in an
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1499 # unexpected format.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1500 # proxies already set up to be empty so nothing to do
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1501 pass
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1502 return proxies
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1503
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1504 def getproxies():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1505 """Return a dictionary of scheme -> proxy server URL mappings.
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1506
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1507 Returns settings gathered from the environment, if specified,
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1508 or the registry.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1509
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1510 """
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1511 return getproxies_environment() or getproxies_registry()
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1512
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1513 def proxy_bypass_registry(host):
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1514 try:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1515 import _winreg
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1516 import re
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1517 except ImportError:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1518 # Std modules, so should be around - but you never know!
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1519 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1520 try:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1521 internetSettings = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER,
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1522 r'Software\Microsoft\Windows\CurrentVersion\Internet Settings')
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1523 proxyEnable = _winreg.QueryValueEx(internetSettings,
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1524 'ProxyEnable')[0]
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1525 proxyOverride = str(_winreg.QueryValueEx(internetSettings,
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1526 'ProxyOverride')[0])
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1527 # ^^^^ Returned as Unicode but problems if not converted to ASCII
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1528 except WindowsError:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1529 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1530 if not proxyEnable or not proxyOverride:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1531 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1532 # try to make a host list from name and IP address.
36576 33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1533 rawHost, port = splitport(host)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1534 host = [rawHost]
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1535 try:
36576 33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1536 addr = socket.gethostbyname(rawHost)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1537 if addr != rawHost:
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1538 host.append(addr)
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1539 except socket.error:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1540 pass
36576 33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1541 try:
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1542 fqdn = socket.getfqdn(rawHost)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1543 if fqdn != rawHost:
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1544 host.append(fqdn)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1545 except socket.error:
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1546 pass
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1547 # make a check value list from the registry entry: replace the
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1548 # '<local>' string by the localhost entry and the corresponding
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1549 # canonical entry.
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1550 proxyOverride = proxyOverride.split(';')
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1551 # now check if we match one of the registry values.
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1552 for test in proxyOverride:
53636 29656a03cce2 Fix for Issue1648102, based on the MSDN spec: If this parameter specifies the Senthil Kumaran <orsenthil@gmail.com> parents: 53297 diff changeset	1553 if test == '<local>':
29656a03cce2 Fix for Issue1648102, based on the MSDN spec: If this parameter specifies the Senthil Kumaran <orsenthil@gmail.com> parents: 53297 diff changeset	1554 if '.' not in rawHost:
29656a03cce2 Fix for Issue1648102, based on the MSDN spec: If this parameter specifies the Senthil Kumaran <orsenthil@gmail.com> parents: 53297 diff changeset	1555 return 1
19119 600a9a2b7a18 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 19104 diff changeset	1556 test = test.replace(".", r"\.") # mask dots
600a9a2b7a18 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 19104 diff changeset	1557 test = test.replace("", r".") # change glob sequence
600a9a2b7a18 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 19104 diff changeset	1558 test = test.replace("?", r".") # change glob char
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1559 for val in host:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1560 # print "%s <--> %s" %( test, val )
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1561 if re.match(test, val, re.I):
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1562 return 1
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1563 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1564
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1565 def proxy_bypass(host):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1566 """Return a dictionary of scheme -> proxy server URL mappings.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1567
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1568 Returns settings gathered from the environment, if specified,
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1569 or the registry.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1570
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1571 """
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1572 if getproxies_environment():
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1573 return proxy_bypass_environment(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1574 else:
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1575 return proxy_bypass_registry(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1576
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1577 else:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1578 # By default use environment variables
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1579 getproxies = getproxies_environment
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1580 proxy_bypass = proxy_bypass_environment
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1581
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1582 # Test and time quote() and unquote()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1583 def test1():
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1584 s = ''
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1585 for i in range(256): s = s + chr(i)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1586 s = s*4
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1587 t0 = time.time()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1588 qs = quote(s)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1589 uqs = unquote(qs)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1590 t1 = time.time()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1591 if uqs != s:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1592 print 'Wrong!'
31515 71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	1593 print repr(s)
71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	1594 print repr(qs)
71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	1595 print repr(uqs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1596 print round(t1 - t0, 3), 'sec'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1597
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1598
9148 e60055216b8f Changes by Eric Raymond: Guido van Rossum <guido@python.org> parents: 8848 diff changeset	1599 def reporthook(blocknum, blocksize, totalsize):
e60055216b8f Changes by Eric Raymond: Guido van Rossum <guido@python.org> parents: 8848 diff changeset	1600 # Report during remote transfers
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1601 print "Block number: %d, Block size: %d, Total size: %d" % (
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1602 blocknum, blocksize, totalsize)

Mercurial > cpython-lru_cache-weakref

annotate Lib/urllib.py @ 79675:c0a2294de787 2.7