cpython-lru_cache-weakref: Lib/urllib.py annotate

annotate Lib/urllib.py @ 54294:dba767bc3155 legacy-trunk

remove mention of old ctypes version

author	Benjamin Peterson <benjamin@python.org>
date	Sat, 23 May 2009 20:59:09 +0000
parents	29656a03cce2
children	2ddddc641a98

rev	line source
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1 """Open an arbitrary URL.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	2
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	3 See the following document for more info on URLs:
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	4 "Names and Addresses, URIs, URLs, URNs, URCs", at
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	5 http://www.w3.org/pub/WWW/Addressing/Overview.html
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	6
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	7 See also the HTTP spec (from which the error codes are derived):
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	8 "HTTP - Hypertext Transfer Protocol", at
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	9 http://www.w3.org/pub/WWW/Protocols/
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	10
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	11 Related standards and specs:
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	12 - RFC1808: the "relative URL" spec. (authoritative status)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	13 - RFC1738 - the "URL standard". (authoritative status)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	14 - RFC1630 - the "URI spec". (informational status)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	15
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	16 The object returned by URLopener().open(file) will differ per
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	17 protocol. All you know is that is has methods read(), readline(),
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	18 readlines(), fileno(), close() and info(). The read*(), fileno()
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	19 and close() methods work like those of open files.
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	20 The info() method returns a mimetools.Message object which can be
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	21 used to query various info about the object, if available.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	22 (mimetools.Message objects are queried with the getheader() method.)
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	23 """
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	24
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	25 import string
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	26 import socket
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	27 import os
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	28 import time
4558 df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	29 import sys
31788 2024b4e6e38d Replace code in urllib for basejoin (undocumented) with urlparse.urljoin . Brett Cannon <bcannon@gmail.com> parents: 31515 diff changeset	30 from urlparse import urljoin as basejoin
47871 801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	31 import warnings
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	32
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16833 diff changeset	33 __all__ = ["urlopen", "URLopener", "FancyURLopener", "urlretrieve",
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16833 diff changeset	34 "urlcleanup", "quote", "quote_plus", "unquote", "unquote_plus",
17039 5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	35 "urlencode", "url2pathname", "pathname2url", "splittag",
5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	36 "localhost", "thishost", "ftperrors", "basejoin", "unwrap",
5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	37 "splittype", "splithost", "splituser", "splitpasswd", "splitport",
5d68afc5227c updated __all__ to include several other names Skip Montanaro <skip@pobox.com> parents: 16983 diff changeset	38 "splitnport", "splitquery", "splitattr", "splitvalue",
41743 2ecc296443c3 Remove the gopherlib module. It has been raising a DeprecationWarning since Brett Cannon <bcannon@gmail.com> parents: 41246 diff changeset	39 "getproxies"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16833 diff changeset	40
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	41 __version__ = '1.17' # XXX This version is not always updated :-(
5727 e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	42
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	43 MAXFTPCACHE = 10 # Trim the ftp cache beyond this size
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	44
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	45 # Helper for non-unix systems
9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	46 if os.name == 'mac':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	47 from macurl2path import url2pathname, pathname2url
7753 8ef067559d5a A few lines were indented using spaces instead of tabs -- fix them. Guido van Rossum <guido@python.org> parents: 7666 diff changeset	48 elif os.name == 'nt':
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	49 from nturl2path import url2pathname, pathname2url
16949 09ba15ee1961 RISCOS changes by dschwertberger. Guido van Rossum <guido@python.org> parents: 16889 diff changeset	50 elif os.name == 'riscos':
09ba15ee1961 RISCOS changes by dschwertberger. Guido van Rossum <guido@python.org> parents: 16889 diff changeset	51 from rourl2path import url2pathname, pathname2url
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	52 else:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	53 def url2pathname(pathname):
36124 536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	54 """OS-specific conversion from a relative URL of the 'file' scheme
536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	55 to a file system path; not recommended for general use."""
10383 70dd91c21513 From: Sjoerd Mullender Guido van Rossum <guido@python.org> parents: 10366 diff changeset	56 return unquote(pathname)
36124 536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	57
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	58 def pathname2url(pathname):
36124 536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	59 """OS-specific conversion from a file system path to a relative URL
536f24e16af2 Bug #649974: make docstrings for url2pathname consistent Georg Brandl <georg@python.org> parents: 35956 diff changeset	60 of the 'file' scheme; not recommended for general use."""
10383 70dd91c21513 From: Sjoerd Mullender Guido van Rossum <guido@python.org> parents: 10366 diff changeset	61 return quote(pathname)
9649 47012c13efe4 Sjoerd Mullender: Guido van Rossum <guido@python.org> parents: 9148 diff changeset	62
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	63 # This really consists of two pieces:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	64 # (1) a class which handles opening of all sorts of URLs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	65 # (plus assorted utilities etc.)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	66 # (2) a set of functions for parsing URLs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	67 # XXX Should these be separated out into different modules?
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	68
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	69
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	70 # Shortcut for basic usage
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	71 _urlopener = None
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	72 def urlopen(url, data=None, proxies=None):
47871 801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	73 """Create a file-like object for the specified URL to read from."""
801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	74 from warnings import warnpy3k
801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	75 warnings.warnpy3k("urllib.urlopen() has been removed in Python 3.0 in "
801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	76 "favor of urllib2.urlopen()", stacklevel=2)
801754201b11 Handle urllib's renaming for Python 3.0: Brett Cannon <bcannon@gmail.com> parents: 47372 diff changeset	77
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	78 global _urlopener
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	79 if proxies is not None:
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	80 opener = FancyURLopener(proxies=proxies)
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	81 elif not _urlopener:
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	82 opener = FancyURLopener()
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	83 _urlopener = opener
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	84 else:
c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	85 opener = _urlopener
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	86 if data is None:
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	87 return opener.open(url)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	88 else:
22822 c56eaad3a0b3 Support manual proxy configuration for simple urlopen() operations. Fred Drake <fdrake@acm.org> parents: 22768 diff changeset	89 return opener.open(url, data)
14151 72d8a3b203b1 Randall Hopper <aa8vb@yahoo.com>>: Fred Drake <fdrake@acm.org> parents: 14118 diff changeset	90 def urlretrieve(url, filename=None, reporthook=None, data=None):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	91 global _urlopener
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	92 if not _urlopener:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	93 _urlopener = FancyURLopener()
14151 72d8a3b203b1 Randall Hopper <aa8vb@yahoo.com>>: Fred Drake <fdrake@acm.org> parents: 14118 diff changeset	94 return _urlopener.retrieve(url, filename, reporthook, data)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	95 def urlcleanup():
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	96 if _urlopener:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	97 _urlopener.cleanup()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	98
43008 4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	99 # check for SSL
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	100 try:
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	101 import ssl
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	102 except:
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	103 _have_ssl = False
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	104 else:
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	105 _have_ssl = True
4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	106
35425 d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	107 # exception raised when downloaded size does not match content-length
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	108 class ContentTooShortError(IOError):
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	109 def __init__(self, message, content):
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	110 IOError.__init__(self, message)
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	111 self.content = content
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	112
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	113 ftpcache = {}
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	114 class URLopener:
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	115 """Class to open URLs.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	116 This is a class rather than just a subroutine because we may need
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	117 more than one set of global protocol-specific options.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	118 Note -- this is a base class for those who don't want the
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	119 automatic handling of errors type 302 (relocated) and 401
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	120 (authorization needed)."""
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	121
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	122 __tempfiles = None
4566 6403c94de50d Added safeguard against failure in __del__. Guido van Rossum <guido@python.org> parents: 4558 diff changeset	123
14157 e7c8b598ff75 Promote the server version from a local variable to a class variable, Guido van Rossum <guido@python.org> parents: 14151 diff changeset	124 version = "Python-urllib/%s" % __version__
e7c8b598ff75 Promote the server version from a local variable to a class variable, Guido van Rossum <guido@python.org> parents: 14151 diff changeset	125
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	126 # Constructor
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	127 def __init__(self, proxies=None, **x509):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	128 if proxies is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	129 proxies = getproxies()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	130 assert hasattr(proxies, 'has_key'), "proxies must be a mapping"
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	131 self.proxies = proxies
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	132 self.key_file = x509.get('key_file')
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	133 self.cert_file = x509.get('cert_file')
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	134 self.addheaders = [('User-Agent', self.version)]
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	135 self.__tempfiles = []
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	136 self.__unlink = os.unlink # See cleanup()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	137 self.tempcache = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	138 # Undocumented feature: if you assign {} to tempcache,
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	139 # it is used to cache files retrieved with
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	140 # self.retrieve(). This is not enabled by default
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	141 # since it does not work for changing documents (and I
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	142 # haven't got the logic to check expiration headers
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	143 # yet).
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	144 self.ftpcache = ftpcache
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	145 # Undocumented feature: you can use a different
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	146 # ftp cache by assigning to the .ftpcache member;
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	147 # in case you want logically independent URL openers
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	148 # XXX This is not threadsafe. Bah.
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	149
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	150 def __del__(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	151 self.close()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	152
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	153 def close(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	154 self.cleanup()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	155
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	156 def cleanup(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	157 # This code sometimes runs when the rest of this module
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	158 # has already been deleted, so it can't use any globals
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	159 # or import anything.
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	160 if self.__tempfiles:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	161 for file in self.__tempfiles:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	162 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	163 self.__unlink(file)
19172 cb346baedad6 Only catch the errors that can actually occur, as reported in bug #411881. Martin v. Löwis <martin@v.loewis.de> parents: 19119 diff changeset	164 except OSError:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	165 pass
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	166 del self.__tempfiles[:]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	167 if self.tempcache:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	168 self.tempcache.clear()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	169
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	170 def addheader(self, *args):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	171 """Add a header to be used by the HTTP interface only
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	172 e.g. u.addheader('Accept', 'sound/basic')"""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	173 self.addheaders.append(args)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	174
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	175 # External interface
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	176 def open(self, fullurl, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	177 """Use URLopener().open(file) instead of open(file, 'r')."""
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	178 fullurl = unwrap(toBytes(fullurl))
53297 c8a75710ab25 Fix for the Issue918368 - urllib doesn't correct server returned urls Senthil Kumaran <orsenthil@gmail.com> parents: 52476 diff changeset	179 # percent encode url, fixing lame server errors for e.g, like space
c8a75710ab25 Fix for the Issue918368 - urllib doesn't correct server returned urls Senthil Kumaran <orsenthil@gmail.com> parents: 52476 diff changeset	180 # within url paths.
c8a75710ab25 Fix for the Issue918368 - urllib doesn't correct server returned urls Senthil Kumaran <orsenthil@gmail.com> parents: 52476 diff changeset	181 fullurl = quote(fullurl, safe="%/:=&?~#+!$,;'@()*[]")
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	182 if self.tempcache and fullurl in self.tempcache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	183 filename, headers = self.tempcache[fullurl]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	184 fp = open(filename, 'rb')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	185 return addinfourl(fp, headers, fullurl)
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	186 urltype, url = splittype(fullurl)
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	187 if not urltype:
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	188 urltype = 'file'
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	189 if urltype in self.proxies:
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	190 proxy = self.proxies[urltype]
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	191 urltype, proxyhost = splittype(proxy)
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	192 host, selector = splithost(proxyhost)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	193 url = (host, fullurl) # Signal special case to open_*()
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	194 else:
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	195 proxy = None
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	196 name = 'open_' + urltype
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	197 self.type = urltype
31794 f1d44295ebf7 Replace sequential split/join calls on strings with a single replace call. Brett Cannon <bcannon@gmail.com> parents: 31788 diff changeset	198 name = name.replace('-', '_')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	199 if not hasattr(self, name):
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	200 if proxy:
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	201 return self.open_unknown_proxy(proxy, fullurl, data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	202 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	203 return self.open_unknown(fullurl, data)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	204 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	205 if data is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	206 return getattr(self, name)(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	207 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	208 return getattr(self, name)(url, data)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	209 except socket.error, msg:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	210 raise IOError, ('socket error', msg), sys.exc_info()[2]
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	211
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	212 def open_unknown(self, fullurl, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	213 """Overridable interface to open unknown URL type."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	214 type, url = splittype(fullurl)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	215 raise IOError, ('url error', 'unknown url type', type)
3028 7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	216
14981 82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	217 def open_unknown_proxy(self, proxy, fullurl, data=None):
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	218 """Overridable interface to open unknown URL type."""
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	219 type, url = splittype(fullurl)
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	220 raise IOError, ('url error', 'invalid proxy for %s' % type, proxy)
82123ce02684 Provide a clearer error message when urlopen fails because of an Jeremy Hylton <jeremy@alum.mit.edu> parents: 14803 diff changeset	221
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	222 # External interface
14181 89f8153fd1ba Pass data on to retrieve method. Sjoerd Mullender <sjoerd@acm.org> parents: 14157 diff changeset	223 def retrieve(self, url, filename=None, reporthook=None, data=None):
28757 99e82dbaf0cc Fix docstring for URLOpener.retrieve() in regards to opening a local file Brett Cannon <bcannon@gmail.com> parents: 28239 diff changeset	224 """retrieve(url) returns (filename, headers) for a local object
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	225 or (tempfilename, headers) for a remote object."""
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	226 url = unwrap(toBytes(url))
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	227 if self.tempcache and url in self.tempcache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	228 return self.tempcache[url]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	229 type, url1 = splittype(url)
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	230 if filename is None and (not type or type == 'file'):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	231 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	232 fp = self.open_local_file(url1)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	233 hdrs = fp.info()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	234 del fp
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	235 return url2pathname(splithost(url1)[1]), hdrs
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	236 except IOError, msg:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	237 pass
14151 72d8a3b203b1 Randall Hopper <aa8vb@yahoo.com>>: Fred Drake <fdrake@acm.org> parents: 14118 diff changeset	238 fp = self.open(url, data)
52265 efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	239 try:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	240 headers = fp.info()
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	241 if filename:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	242 tfp = open(filename, 'wb')
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	243 else:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	244 import tempfile
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	245 garbage, path = splittype(url)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	246 garbage, path = splithost(path or "")
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	247 path, garbage = splitquery(path or "")
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	248 path, garbage = splitattr(path or "")
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	249 suffix = os.path.splitext(path)[1]
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	250 (fd, filename) = tempfile.mkstemp(suffix)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	251 self.__tempfiles.append(filename)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	252 tfp = os.fdopen(fd, 'wb')
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	253 try:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	254 result = filename, headers
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	255 if self.tempcache is not None:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	256 self.tempcache[url] = result
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	257 bs = 1024*8
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	258 size = -1
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	259 read = 0
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	260 blocknum = 0
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	261 if reporthook:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	262 if "content-length" in headers:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	263 size = int(headers["Content-Length"])
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	264 reporthook(blocknum, bs, size)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	265 while 1:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	266 block = fp.read(bs)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	267 if block == "":
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	268 break
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	269 read += len(block)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	270 tfp.write(block)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	271 blocknum += 1
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	272 if reporthook:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	273 reporthook(blocknum, bs, size)
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	274 finally:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	275 tfp.close()
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	276 finally:
efd17cb5823a close the file even if an exception occurs #5536 Benjamin Peterson <benjamin@python.org> parents: 50673 diff changeset	277 fp.close()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	278 del fp
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	279 del tfp
35425 d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	280
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	281 # raise exception if actual size does not match content-length header
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	282 if size >= 0 and read < size:
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	283 raise ContentTooShortError("retrieval incomplete: got only %i out "
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	284 "of %i bytes" % (read, size), result)
d5df5222fd6d Patch [ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld Georg Brandl <georg@python.org> parents: 34293 diff changeset	285
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	286 return result
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	287
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	288 # Each method named open_<type> knows how to open that type of URL
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	289
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	290 def open_http(self, url, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	291 """Use HTTP protocol."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	292 import httplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	293 user_passwd = None
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	294 proxy_passwd= None
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	295 if isinstance(url, str):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	296 host, selector = splithost(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	297 if host:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	298 user_passwd, host = splituser(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	299 host = unquote(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	300 realhost = host
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	301 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	302 host, selector = url
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	303 # check whether the proxy contains authorization information
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	304 proxy_passwd, host = splituser(host)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	305 # now we proceed with the url we want to obtain
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	306 urltype, rest = splittype(selector)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	307 url = rest
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	308 user_passwd = None
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	309 if urltype.lower() != 'http':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	310 realhost = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	311 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	312 realhost, rest = splithost(rest)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	313 if realhost:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	314 user_passwd, realhost = splituser(realhost)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	315 if user_passwd:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	316 selector = "%s://%s%s" % (urltype, realhost, rest)
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	317 if proxy_bypass(realhost):
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	318 host = realhost
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	319
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	320 #print "proxy via http:", host, selector
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	321 if not host: raise IOError, ('http error', 'no host given')
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	322
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	323 if proxy_passwd:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	324 import base64
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	325 proxy_auth = base64.b64encode(proxy_passwd).strip()
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	326 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	327 proxy_auth = None
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	328
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	329 if user_passwd:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	330 import base64
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	331 auth = base64.b64encode(user_passwd).strip()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	332 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	333 auth = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	334 h = httplib.HTTP(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	335 if data is not None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	336 h.putrequest('POST', selector)
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	337 h.putheader('Content-Type', 'application/x-www-form-urlencoded')
7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	338 h.putheader('Content-Length', '%d' % len(data))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	339 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	340 h.putrequest('GET', selector)
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	341 if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' % proxy_auth)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	342 if auth: h.putheader('Authorization', 'Basic %s' % auth)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	343 if realhost: h.putheader('Host', realhost)
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	344 for args in self.addheaders: h.putheader(*args)
50673 f1ca5dd59aca Issue 4336: Let users of HTTPConnection.endheaders() submit a message body to the function if required. Kristján Valur Jónsson <kristjan@ccpgames.com> parents: 49260 diff changeset	345 h.endheaders(data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	346 errcode, errmsg, headers = h.getreply()
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	347 fp = h.getfile()
41186 c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	348 if errcode == -1:
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	349 if fp: fp.close()
41186 c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	350 # something went wrong with the HTTP status line
c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	351 raise IOError, ('http protocol error', 0,
c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	352 'got a bad status line', None)
43404 d1fa820b2dca issue1177: Ported Facundo's from urllib2 to urllib, accepting 2xx responses. Sean Reifscheider <jafo@tummy.com> parents: 43008 diff changeset	353 # According to RFC 2616, "2xx" code indicates that the client's
d1fa820b2dca issue1177: Ported Facundo's from urllib2 to urllib, accepting 2xx responses. Sean Reifscheider <jafo@tummy.com> parents: 43008 diff changeset	354 # request was successfully received, understood, and accepted.
44423 099a17e2d2bd Issue1177 Kurt B. Kaiser <kbk@shore.net> parents: 43433 diff changeset	355 if (200 <= errcode < 300):
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	356 return addinfourl(fp, headers, "http:" + url, errcode)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	357 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	358 if data is None:
10366 65e7212df1bc open_http also had the 'data is None' test backwards. don't call with the Guido van Rossum <guido@python.org> parents: 10308 diff changeset	359 return self.http_error(url, fp, errcode, errmsg, headers)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	360 else:
10366 65e7212df1bc open_http also had the 'data is None' test backwards. don't call with the Guido van Rossum <guido@python.org> parents: 10308 diff changeset	361 return self.http_error(url, fp, errcode, errmsg, headers, data)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	362
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	363 def http_error(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	364 """Handle http errors.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	365 Derived class can override this, or provide specific handlers
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	366 named http_error_DDD where DDD is the 3-digit error code."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	367 # First check if there's a specific handler for this error
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	368 name = 'http_error_%d' % errcode
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	369 if hasattr(self, name):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	370 method = getattr(self, name)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	371 if data is None:
10308 e30893961398 http_error had the 'data is None' test backwards. don't call with the Jeremy Hylton <jeremy@alum.mit.edu> parents: 10307 diff changeset	372 result = method(url, fp, errcode, errmsg, headers)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	373 else:
10308 e30893961398 http_error had the 'data is None' test backwards. don't call with the Jeremy Hylton <jeremy@alum.mit.edu> parents: 10307 diff changeset	374 result = method(url, fp, errcode, errmsg, headers, data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	375 if result: return result
10308 e30893961398 http_error had the 'data is None' test backwards. don't call with the Jeremy Hylton <jeremy@alum.mit.edu> parents: 10307 diff changeset	376 return self.http_error_default(url, fp, errcode, errmsg, headers)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	377
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	378 def http_error_default(self, url, fp, errcode, errmsg, headers):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	379 """Default error handler: close the connection and raise IOError."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	380 void = fp.read()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	381 fp.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	382 raise IOError, ('http error', errcode, errmsg, headers)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	383
43008 4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	384 if _have_ssl:
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	385 def open_https(self, url, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	386 """Use HTTPS protocol."""
43008 4771639fc049 This contains a number of things: Bill Janssen <janssen@parc.com> parents: 41854 diff changeset	387
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	388 import httplib
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	389 user_passwd = None
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	390 proxy_passwd = None
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	391 if isinstance(url, str):
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	392 host, selector = splithost(url)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	393 if host:
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	394 user_passwd, host = splituser(host)
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	395 host = unquote(host)
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	396 realhost = host
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	397 else:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	398 host, selector = url
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	399 # here, we determine, whether the proxy contains authorization information
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	400 proxy_passwd, host = splituser(host)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	401 urltype, rest = splittype(selector)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	402 url = rest
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	403 user_passwd = None
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	404 if urltype.lower() != 'https':
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	405 realhost = None
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	406 else:
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	407 realhost, rest = splithost(rest)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	408 if realhost:
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	409 user_passwd, realhost = splituser(realhost)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	410 if user_passwd:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	411 selector = "%s://%s%s" % (urltype, realhost, rest)
12905 25431972294b Comment out an apparent debug print Andrew M. Kuchling <amk@amk.ca> parents: 12741 diff changeset	412 #print "proxy via https:", host, selector
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	413 if not host: raise IOError, ('https error', 'no host given')
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	414 if proxy_passwd:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	415 import base64
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	416 proxy_auth = base64.b64encode(proxy_passwd).strip()
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	417 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	418 proxy_auth = None
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	419 if user_passwd:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	420 import base64
40205 9d691d3755d0 [Patch #1574068 by Scott Dial] urllib and urllib2 were using Andrew M. Kuchling <amk@amk.ca> parents: 39188 diff changeset	421 auth = base64.b64encode(user_passwd).strip()
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	422 else:
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	423 auth = None
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	424 h = httplib.HTTPS(host, 0,
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	425 key_file=self.key_file,
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	426 cert_file=self.cert_file)
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	427 if data is not None:
144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	428 h.putrequest('POST', selector)
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	429 h.putheader('Content-Type',
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	430 'application/x-www-form-urlencoded')
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	431 h.putheader('Content-Length', '%d' % len(data))
12462 144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	432 else:
144a89570a39 Fixed bug reported by JP Calderone: https:// URL's didn't work. Andrew M. Kuchling <amk@amk.ca> parents: 11707 diff changeset	433 h.putrequest('GET', selector)
40476 dcc40b93bfbe [Patch #1617413 from Dug Song] Fix HTTP Basic authentication via HTTPS Andrew M. Kuchling <amk@amk.ca> parents: 40205 diff changeset	434 if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' % proxy_auth)
dcc40b93bfbe [Patch #1617413 from Dug Song] Fix HTTP Basic authentication via HTTPS Andrew M. Kuchling <amk@amk.ca> parents: 40205 diff changeset	435 if auth: h.putheader('Authorization', 'Basic %s' % auth)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	436 if realhost: h.putheader('Host', realhost)
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	437 for args in self.addheaders: h.putheader(*args)
50673 f1ca5dd59aca Issue 4336: Let users of HTTPConnection.endheaders() submit a message body to the function if required. Kristján Valur Jónsson <kristjan@ccpgames.com> parents: 49260 diff changeset	438 h.endheaders(data)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	439 errcode, errmsg, headers = h.getreply()
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	440 fp = h.getfile()
41186 c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	441 if errcode == -1:
41246 b8b17a2face5 Try to get test_urllib to pass on Windows by closing the file. Neal Norwitz <nnorwitz@gmail.com> parents: 41186 diff changeset	442 if fp: fp.close()
41186 c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	443 # something went wrong with the HTTP status line
c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	444 raise IOError, ('http protocol error', 0,
c301b5bcdfdc Bug #767111: fix long-standing bug in urllib which caused an Georg Brandl <georg@python.org> parents: 41116 diff changeset	445 'got a bad status line', None)
43433 773e976bab3a #1177: accept 2xx responses for https too, not only http. Georg Brandl <georg@python.org> parents: 43404 diff changeset	446 # According to RFC 2616, "2xx" code indicates that the client's
773e976bab3a #1177: accept 2xx responses for https too, not only http. Georg Brandl <georg@python.org> parents: 43404 diff changeset	447 # request was successfully received, understood, and accepted.
44423 099a17e2d2bd Issue1177 Kurt B. Kaiser <kbk@shore.net> parents: 43433 diff changeset	448 if (200 <= errcode < 300):
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	449 return addinfourl(fp, headers, "https:" + url, errcode)
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	450 else:
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	451 if data is None:
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	452 return self.http_error(url, fp, errcode, errmsg, headers)
17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	453 else:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	454 return self.http_error(url, fp, errcode, errmsg, headers,
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	455 data)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	456
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	457 def open_file(self, url):
37546 edc6a0cc8466 Fix a couple of strings that were no-ops. urllib.open_file was a docstring Neal Norwitz <nnorwitz@gmail.com> parents: 37287 diff changeset	458 """Use local file or FTP depending on form of URL."""
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	459 if not isinstance(url, str):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	460 raise IOError, ('file error', 'proxy support for file protocol currently not implemented')
25240 c8044123d110 When testing for localhost/ first map to lower case. Spotted by Skip. Jack Jansen <jack.jansen@cwi.nl> parents: 25239 diff changeset	461 if url[:2] == '//' and url[2:3] != '/' and url[2:12].lower() != 'localhost/':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	462 return self.open_ftp(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	463 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	464 return self.open_local_file(url)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	465
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	466 def open_local_file(self, url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	467 """Use local file."""
40689 b23711be2b3e Use new email module names (#1637162, #1637159, #1637157). Georg Brandl <georg@python.org> parents: 40476 diff changeset	468 import mimetypes, mimetools, email.utils
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	469 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	470 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	471 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	472 from StringIO import StringIO
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	473 host, file = splithost(url)
3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	474 localname = url2pathname(file)
23034 53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	475 try:
53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	476 stats = os.stat(localname)
53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	477 except OSError, e:
53ba9110a1f1 Fix from SF bug #541980 (Jacques A. Vidrine). Guido van Rossum <guido@python.org> parents: 22822 diff changeset	478 raise IOError(e.errno, e.strerror, e.filename)
22458 68fe2b391159 use stat attributes instead of tuple entries Walter Dörwald <walter@livinglogic.de> parents: 22401 diff changeset	479 size = stats.st_size
40689 b23711be2b3e Use new email module names (#1637162, #1637159, #1637157). Georg Brandl <georg@python.org> parents: 40476 diff changeset	480 modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	481 mtype = mimetypes.guess_type(url)[0]
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	482 headers = mimetools.Message(StringIO(
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	483 'Content-Type: %s\nContent-Length: %d\nLast-modified: %s\n' %
3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	484 (mtype or 'text/plain', size, modified)))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	485 if not host:
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	486 urlfile = file
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	487 if file[:1] == '/':
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	488 urlfile = 'file://' + file
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	489 return addinfourl(open(localname, 'rb'),
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	490 headers, urlfile)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	491 host, port = splitport(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	492 if not port \
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	493 and socket.gethostbyname(host) in (localhost(), thishost()):
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	494 urlfile = file
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	495 if file[:1] == '/':
fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	496 urlfile = 'file://' + file
19103 3420c9e6af11 SF Patch #420725 by Walter Doerwald: Guido van Rossum <guido@python.org> parents: 18378 diff changeset	497 return addinfourl(open(localname, 'rb'),
11046 fd546cbdfb5f Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 10943 diff changeset	498 headers, urlfile)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	499 raise IOError, ('local file error', 'not on local host')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	500
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	501 def open_ftp(self, url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	502 """Use FTP protocol."""
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	503 if not isinstance(url, str):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	504 raise IOError, ('ftp error', 'proxy support for ftp protocol currently not implemented')
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	505 import mimetypes, mimetools
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	506 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	507 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	508 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	509 from StringIO import StringIO
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	510 host, path = splithost(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	511 if not host: raise IOError, ('ftp error', 'no host given')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	512 host, port = splitport(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	513 user, host = splituser(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	514 if user: user, passwd = splitpasswd(user)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	515 else: passwd = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	516 host = unquote(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	517 user = unquote(user or '')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	518 passwd = unquote(passwd or '')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	519 host = socket.gethostbyname(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	520 if not port:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	521 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	522 port = ftplib.FTP_PORT
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	523 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	524 port = int(port)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	525 path, attrs = splitattr(path)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	526 path = unquote(path)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	527 dirs = path.split('/')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	528 dirs, file = dirs[:-1], dirs[-1]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	529 if dirs and not dirs[0]: dirs = dirs[1:]
11201 bf565b18cfdc Patches by Michael Reilly to correctly deal with ftp URLs of the form Guido van Rossum <guido@python.org> parents: 11094 diff changeset	530 if dirs and not dirs[0]: dirs[0] = '/'
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	531 key = user, host, port, '/'.join(dirs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	532 # XXX thread unsafe!
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	533 if len(self.ftpcache) > MAXFTPCACHE:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	534 # Prune the cache, rather arbitrarily
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	535 for k in self.ftpcache.keys():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	536 if k != key:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	537 v = self.ftpcache[k]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	538 del self.ftpcache[k]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	539 v.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	540 try:
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	541 if not key in self.ftpcache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	542 self.ftpcache[key] = \
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	543 ftpwrapper(user, passwd, host, port, dirs)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	544 if not file: type = 'D'
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	545 else: type = 'I'
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	546 for attr in attrs:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	547 attr, value = splitvalue(attr)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	548 if attr.lower() == 'type' and \
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	549 value in ('a', 'A', 'i', 'I', 'd', 'D'):
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	550 type = value.upper()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	551 (fp, retrlen) = self.ftpcache[key].retrfile(file, type)
19440 ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	552 mtype = mimetypes.guess_type("ftp:" + url)[0]
ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	553 headers = ""
ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	554 if mtype:
ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	555 headers += "Content-Type: %s\n" % mtype
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	556 if retrlen is not None and retrlen >= 0:
19440 ca1b8cc1ba2b SF patch #454553 by Walter Dörwald: auto-guess content-type header for Guido van Rossum <guido@python.org> parents: 19172 diff changeset	557 headers += "Content-Length: %d\n" % retrlen
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	558 headers = mimetools.Message(StringIO(headers))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	559 return addinfourl(fp, headers, "ftp:" + url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	560 except ftperrors(), msg:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	561 raise IOError, ('ftp error', msg), sys.exc_info()[2]
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	562
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	563 def open_data(self, url, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	564 """Use "data" URL."""
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	565 if not isinstance(url, str):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	566 raise IOError, ('data error', 'proxy support for data protocol currently not implemented')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	567 # ignore POSTed data
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	568 #
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	569 # syntax of data URLs:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	570 # dataurl := "data:" [ mediatype ] [ ";base64" ] "," data
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	571 # mediatype := [ type "/" subtype ] *( ";" parameter )
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	572 # data := *urlchar
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	573 # parameter := attribute "=" value
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	574 import mimetools
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	575 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	576 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	577 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	578 from StringIO import StringIO
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	579 try:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	580 [type, data] = url.split(',', 1)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	581 except ValueError:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	582 raise IOError, ('data error', 'bad data URL')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	583 if not type:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	584 type = 'text/plain;charset=US-ASCII'
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	585 semi = type.rfind(';')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	586 if semi >= 0 and '=' not in type[semi:]:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	587 encoding = type[semi+1:]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	588 type = type[:semi]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	589 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	590 encoding = ''
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	591 msg = []
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	592 msg.append('Date: %s'%time.strftime('%a, %d %b %Y %T GMT',
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	593 time.gmtime(time.time())))
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	594 msg.append('Content-type: %s' % type)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	595 if encoding == 'base64':
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	596 import base64
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	597 data = base64.decodestring(data)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	598 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	599 data = unquote(data)
39188 7da6f81ad4d5 Bug #1459963: properly capitalize HTTP header names. Georg Brandl <georg@python.org> parents: 38053 diff changeset	600 msg.append('Content-Length: %d' % len(data))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	601 msg.append('')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	602 msg.append(data)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	603 msg = '\n'.join(msg)
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	604 f = StringIO(msg)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	605 headers = mimetools.Message(f, 0)
35956 75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	606 #f.fileno = None # needed for addinfourl
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	607 return addinfourl(f, headers, url)
7666 c9394aeaac6a Added support for "data" URL, by Sjoerd Mullender. Guido van Rossum <guido@python.org> parents: 7272 diff changeset	608
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	609
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	610 class FancyURLopener(URLopener):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	611 """Derived class with handlers for errors we can handle (perhaps)."""
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	612
23781 730490ff25ea Fix SF #565414, FancyURLopener() needs to support *kwargs Neal Norwitz <nnorwitz@gmail.com>* parents: 23645 diff changeset	613 def __init__(self, args, *kwargs):
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	614 URLopener.__init__(self, args, *kwargs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	615 self.auth_cache = {}
16611 e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	616 self.tries = 0
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	617 self.maxtries = 10
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	618
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	619 def http_error_default(self, url, fp, errcode, errmsg, headers):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	620 """Default error handling -- don't raise an exception."""
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	621 return addinfourl(fp, headers, "http:" + url, errcode)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	622
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	623 def http_error_302(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	624 """Error 302 -- relocated (temporarily)."""
16611 e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	625 self.tries += 1
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	626 if self.maxtries and self.tries >= self.maxtries:
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	627 if hasattr(self, "http_error_500"):
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	628 meth = self.http_error_500
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	629 else:
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	630 meth = self.http_error_default
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	631 self.tries = 0
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	632 return meth(url, fp, 500,
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	633 "Internal Server Error: Redirect Recursion", headers)
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	634 result = self.redirect_internal(url, fp, errcode, errmsg, headers,
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	635 data)
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	636 self.tries = 0
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	637 return result
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	638
e3cffc0394bf provide simple recovery/escape from apparent redirect recursion. If the Skip Montanaro <skip@pobox.com> parents: 16535 diff changeset	639 def redirect_internal(self, url, fp, errcode, errmsg, headers, data):
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	640 if 'location' in headers:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	641 newurl = headers['location']
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	642 elif 'uri' in headers:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	643 newurl = headers['uri']
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	644 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	645 return
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	646 void = fp.read()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	647 fp.close()
10497 8d40e6ba4c9e Hack so that if a 302 or 301 redirect contains a relative URL, the Guido van Rossum <guido@python.org> parents: 10398 diff changeset	648 # In case the server sent a relative URL, join with original:
17350 3308afcaa131 fixing 408085 - redirect from https becomes http Moshe Zadka <moshez@math.huji.ac.il> parents: 17039 diff changeset	649 newurl = basejoin(self.type + ":" + url, newurl)
29125 92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	650 return self.open(newurl)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	651
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	652 def http_error_301(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	653 """Error 301 -- also relocated (permanently)."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	654 return self.http_error_302(url, fp, errcode, errmsg, headers, data)
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	655
28771 29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	656 def http_error_303(self, url, fp, errcode, errmsg, headers, data=None):
29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	657 """Error 303 -- also relocated (essentially identical to 302)."""
29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	658 return self.http_error_302(url, fp, errcode, errmsg, headers, data)
29b6a48f4c45 SF Patch 549151: urllib2 POSTs on redirect Raymond Hettinger <python@rcn.com> parents: 28757 diff changeset	659
29125 92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	660 def http_error_307(self, url, fp, errcode, errmsg, headers, data=None):
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	661 """Error 307 -- relocated, but turn POST into error."""
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	662 if data is None:
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	663 return self.http_error_302(url, fp, errcode, errmsg, headers, data)
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	664 else:
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	665 return self.http_error_default(url, fp, errcode, errmsg, headers)
92e53c8e7a72 More fixes according to SF 549151: Guido van Rossum <guido@python.org> parents: 28771 diff changeset	666
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	667 def http_error_401(self, url, fp, errcode, errmsg, headers, data=None):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	668 """Error 401 -- authentication required.
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	669 This function supports Basic authentication only."""
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	670 if not 'www-authenticate' in headers:
16833 bbdeb6d4dea1 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16784 diff changeset	671 URLopener.http_error_default(self, url, fp,
20421 ee89567bcab4 Added missing parameter in call to http_error_default(); Fred Drake <fdrake@acm.org> parents: 19504 diff changeset	672 errcode, errmsg, headers)
16784 60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	673 stuff = headers['www-authenticate']
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	674 import re
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	675 match = re.match('[ \t]([^ \t]+)[ \t]+realm="([^"])"', stuff)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	676 if not match:
16833 bbdeb6d4dea1 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16784 diff changeset	677 URLopener.http_error_default(self, url, fp,
16784 60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	678 errcode, errmsg, headers)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	679 scheme, realm = match.groups()
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	680 if scheme.lower() != 'basic':
16833 bbdeb6d4dea1 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16784 diff changeset	681 URLopener.http_error_default(self, url, fp,
16784 60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	682 errcode, errmsg, headers)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	683 name = 'retry_' + self.type + '_basic_auth'
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	684 if data is None:
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	685 return getattr(self,name)(url, realm)
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	686 else:
60fd9d454206 Fixing bug #227562 by calling URLopener.http_error_default when Moshe Zadka <moshez@math.huji.ac.il> parents: 16611 diff changeset	687 return getattr(self,name)(url, realm, data)
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	688
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	689 def http_error_407(self, url, fp, errcode, errmsg, headers, data=None):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	690 """Error 407 -- proxy authentication required.
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	691 This function supports Basic authentication only."""
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	692 if not 'proxy-authenticate' in headers:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	693 URLopener.http_error_default(self, url, fp,
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	694 errcode, errmsg, headers)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	695 stuff = headers['proxy-authenticate']
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	696 import re
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	697 match = re.match('[ \t]([^ \t]+)[ \t]+realm="([^"])"', stuff)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	698 if not match:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	699 URLopener.http_error_default(self, url, fp,
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	700 errcode, errmsg, headers)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	701 scheme, realm = match.groups()
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	702 if scheme.lower() != 'basic':
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	703 URLopener.http_error_default(self, url, fp,
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	704 errcode, errmsg, headers)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	705 name = 'retry_proxy_' + self.type + '_basic_auth'
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	706 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	707 return getattr(self,name)(url, realm)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	708 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	709 return getattr(self,name)(url, realm, data)
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	710
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	711 def retry_proxy_http_basic_auth(self, url, realm, data=None):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	712 host, selector = splithost(url)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	713 newurl = 'http://' + host + selector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	714 proxy = self.proxies['http']
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	715 urltype, proxyhost = splittype(proxy)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	716 proxyhost, proxyselector = splithost(proxyhost)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	717 i = proxyhost.find('@') + 1
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	718 proxyhost = proxyhost[i:]
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	719 user, passwd = self.get_user_passwd(proxyhost, realm, i)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	720 if not (user or passwd): return None
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	721 proxyhost = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + proxyhost
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	722 self.proxies['http'] = 'http://' + proxyhost + proxyselector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	723 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	724 return self.open(newurl)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	725 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	726 return self.open(newurl, data)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	727
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	728 def retry_proxy_https_basic_auth(self, url, realm, data=None):
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	729 host, selector = splithost(url)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	730 newurl = 'https://' + host + selector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	731 proxy = self.proxies['https']
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	732 urltype, proxyhost = splittype(proxy)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	733 proxyhost, proxyselector = splithost(proxyhost)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	734 i = proxyhost.find('@') + 1
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	735 proxyhost = proxyhost[i:]
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	736 user, passwd = self.get_user_passwd(proxyhost, realm, i)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	737 if not (user or passwd): return None
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	738 proxyhost = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + proxyhost
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	739 self.proxies['https'] = 'https://' + proxyhost + proxyselector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	740 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	741 return self.open(newurl)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	742 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	743 return self.open(newurl, data)
36402 bfb247d7bc3f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36398 diff changeset	744
11688 1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	745 def retry_http_basic_auth(self, url, realm, data=None):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	746 host, selector = splithost(url)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	747 i = host.find('@') + 1
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	748 host = host[i:]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	749 user, passwd = self.get_user_passwd(host, realm, i)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	750 if not (user or passwd): return None
15889 b0ba46b16e07 - Make sure to quote the username and password (SF patch #103236 by Guido van Rossum <guido@python.org> parents: 15874 diff changeset	751 host = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + host
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	752 newurl = 'http://' + host + selector
11688 1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	753 if data is None:
1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	754 return self.open(newurl)
1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	755 else:
1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	756 return self.open(newurl, data)
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	757
11688 1c4354bba42b Sjoerd Mullender writes: Guido van Rossum <guido@python.org> parents: 11527 diff changeset	758 def retry_https_basic_auth(self, url, realm, data=None):
15874 4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	759 host, selector = splithost(url)
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	760 i = host.find('@') + 1
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	761 host = host[i:]
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	762 user, passwd = self.get_user_passwd(host, realm, i)
4c01ee0a8073 Whitespace normalization. Top level of Lib now fixed-point for reindent.py! Tim Peters <tim.peters@gmail.com> parents: 15762 diff changeset	763 if not (user or passwd): return None
15889 b0ba46b16e07 - Make sure to quote the username and password (SF patch #103236 by Guido van Rossum <guido@python.org> parents: 15874 diff changeset	764 host = quote(user, safe='') + ':' + quote(passwd, safe='') + '@' + host
36398 d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	765 newurl = 'https://' + host + selector
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	766 if data is None:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	767 return self.open(newurl)
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	768 else:
d68f73f96f31 Patch #1349118: urllib2 now supports user:pass@ style proxy Martin v. Löwis <martin@v.loewis.de> parents: 36124 diff changeset	769 return self.open(newurl, data)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	770
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	771 def get_user_passwd(self, host, realm, clear_cache = 0):
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	772 key = realm + '@' + host.lower()
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 23540 diff changeset	773 if key in self.auth_cache:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	774 if clear_cache:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	775 del self.auth_cache[key]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	776 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	777 return self.auth_cache[key]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	778 user, passwd = self.prompt_user_passwd(host, realm)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	779 if user or passwd: self.auth_cache[key] = (user, passwd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	780 return user, passwd
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	781
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	782 def prompt_user_passwd(self, host, realm):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	783 """Override this in a GUI environment!"""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	784 import getpass
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	785 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	786 user = raw_input("Enter username for %s at %s: " % (realm,
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	787 host))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	788 passwd = getpass.getpass("Enter password for %s in %s at %s: " %
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	789 (user, realm, host))
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	790 return user, passwd
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	791 except KeyboardInterrupt:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	792 print
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	793 return None, None
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	794
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	795
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	796 # Utility functions
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	797
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	798 _localhost = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	799 def localhost():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	800 """Return the IP address of the magic hostname 'localhost'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	801 global _localhost
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	802 if _localhost is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	803 _localhost = socket.gethostbyname('localhost')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	804 return _localhost
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	805
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	806 _thishost = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	807 def thishost():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	808 """Return the IP address of the current host."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	809 global _thishost
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	810 if _thishost is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	811 _thishost = socket.gethostbyname(socket.gethostname())
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	812 return _thishost
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	813
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	814 _ftperrors = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	815 def ftperrors():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	816 """Return the set of errors raised by the FTP class."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	817 global _ftperrors
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	818 if _ftperrors is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	819 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	820 _ftperrors = ftplib.all_errors
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	821 return _ftperrors
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	822
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	823 _noheaders = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	824 def noheaders():
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	825 """Return an empty mimetools.Message object."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	826 global _noheaders
23628 ea979c5206b7 Replaced boolean tests with is None. Raymond Hettinger <python@rcn.com> parents: 23620 diff changeset	827 if _noheaders is None:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	828 import mimetools
34293 cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	829 try:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	830 from cStringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	831 except ImportError:
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	832 from StringIO import StringIO
cd517701fd93 Use cStringIO where available. Raymond Hettinger <python@rcn.com> parents: 33716 diff changeset	833 _noheaders = mimetools.Message(StringIO(), 0)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	834 _noheaders.fp.close() # Recycle file descriptor
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	835 return _noheaders
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	836
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	837
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	838 # Utility classes
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	839
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	840 class ftpwrapper:
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	841 """Class used by open_ftp() for cache of open FTP connections."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	842
47372 e2c5ae30cffe Fixed the semantic of timeout for socket.create_connection and Facundo Batista <facundobatista@gmail.com> parents: 47162 diff changeset	843 def __init__(self, user, passwd, host, port, dirs,
e2c5ae30cffe Fixed the semantic of timeout for socket.create_connection and Facundo Batista <facundobatista@gmail.com> parents: 47162 diff changeset	844 timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	845 self.user = user
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	846 self.passwd = passwd
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	847 self.host = host
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	848 self.port = port
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	849 self.dirs = dirs
41854 10426ad5a6e0 Added an optional timeout parameter to urllib.ftpwrapper, with tests Facundo Batista <facundobatista@gmail.com> parents: 41743 diff changeset	850 self.timeout = timeout
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	851 self.init()
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	852
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	853 def init(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	854 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	855 self.busy = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	856 self.ftp = ftplib.FTP()
41854 10426ad5a6e0 Added an optional timeout parameter to urllib.ftpwrapper, with tests Facundo Batista <facundobatista@gmail.com> parents: 41743 diff changeset	857 self.ftp.connect(self.host, self.port, self.timeout)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	858 self.ftp.login(self.user, self.passwd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	859 for dir in self.dirs:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	860 self.ftp.cwd(dir)
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	861
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	862 def retrfile(self, file, type):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	863 import ftplib
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	864 self.endtransfer()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	865 if type in ('d', 'D'): cmd = 'TYPE A'; isdir = 1
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	866 else: cmd = 'TYPE ' + type; isdir = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	867 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	868 self.ftp.voidcmd(cmd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	869 except ftplib.all_errors:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	870 self.init()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	871 self.ftp.voidcmd(cmd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	872 conn = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	873 if file and not isdir:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	874 # Try to retrieve as a file
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	875 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	876 cmd = 'RETR ' + file
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	877 conn = self.ftp.ntransfercmd(cmd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	878 except ftplib.error_perm, reason:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	879 if str(reason)[:3] != '550':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	880 raise IOError, ('ftp error', reason), sys.exc_info()[2]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	881 if not conn:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	882 # Set transfer mode to ASCII!
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	883 self.ftp.voidcmd('TYPE A')
44819 abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	884 # Try a directory listing. Verify that directory exists.
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	885 if file:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	886 pwd = self.ftp.pwd()
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	887 try:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	888 try:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	889 self.ftp.cwd(file)
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	890 except ftplib.error_perm, reason:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	891 raise IOError, ('ftp error', reason), sys.exc_info()[2]
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	892 finally:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	893 self.ftp.cwd(pwd)
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	894 cmd = 'LIST ' + file
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	895 else:
abcb7923d898 #1664522: in urllib, don't read non-existing directories in ftp mode, Georg Brandl <georg@python.org> parents: 44818 diff changeset	896 cmd = 'LIST'
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	897 conn = self.ftp.ntransfercmd(cmd)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	898 self.busy = 1
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	899 # Pass back both a suitably decorated object and a retrieval length
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	900 return (addclosehook(conn[0].makefile('rb'),
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	901 self.endtransfer), conn[1])
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	902 def endtransfer(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	903 if not self.busy:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	904 return
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	905 self.busy = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	906 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	907 self.ftp.voidresp()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	908 except ftperrors():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	909 pass
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	910
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	911 def close(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	912 self.endtransfer()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	913 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	914 self.ftp.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	915 except ftperrors():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	916 pass
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	917
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	918 class addbase:
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	919 """Base class for addinfo and addclosehook."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	920
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	921 def __init__(self, fp):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	922 self.fp = fp
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	923 self.read = self.fp.read
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	924 self.readline = self.fp.readline
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	925 if hasattr(self.fp, "readlines"): self.readlines = self.fp.readlines
35956 75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	926 if hasattr(self.fp, "fileno"):
75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	927 self.fileno = self.fp.fileno
75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	928 else:
75a79cc4cf7b bug #1365984: urllib and data: URLs. Problem was that cStringIO objects cannot be assigned attributes on the fly. Georg Brandl <georg@python.org> parents: 35723 diff changeset	929 self.fileno = lambda: None
28239 8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	930 if hasattr(self.fp, "__iter__"):
8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	931 self.__iter__ = self.fp.__iter__
8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	932 if hasattr(self.fp, "next"):
8f23d39a9a33 SF 698520: Iterator for urllib.URLOpener Raymond Hettinger <python@rcn.com> parents: 28089 diff changeset	933 self.next = self.fp.next
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	934
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	935 def __repr__(self):
31515 71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	936 return '<%s at %r whose fp = %r>' % (self.__class__.__name__,
71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	937 id(self), self.fp)
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	938
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	939 def close(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	940 self.read = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	941 self.readline = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	942 self.readlines = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	943 self.fileno = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	944 if self.fp: self.fp.close()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	945 self.fp = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	946
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	947 class addclosehook(addbase):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	948 """Class to add a close hook to an open file."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	949
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	950 def __init__(self, fp, closehook, *hookargs):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	951 addbase.__init__(self, fp)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	952 self.closehook = closehook
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	953 self.hookargs = hookargs
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	954
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	955 def close(self):
12741 b50bf7d3961f Fix a problem reported by Oleg Broytmann, who complains that very Guido van Rossum <guido@python.org> parents: 12471 diff changeset	956 addbase.close(self)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	957 if self.closehook:
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 25724 diff changeset	958 self.closehook(*self.hookargs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	959 self.closehook = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	960 self.hookargs = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	961
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	962 class addinfo(addbase):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	963 """class to add an info() method to an open file."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	964
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	965 def __init__(self, fp, headers):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	966 addbase.__init__(self, fp)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	967 self.headers = headers
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	968
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	969 def info(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	970 return self.headers
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	971
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	972 class addinfourl(addbase):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	973 """class to add info() and geturl() methods to an open file."""
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	974
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	975 def __init__(self, fp, headers, url, code=None):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	976 addbase.__init__(self, fp)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	977 self.headers = headers
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	978 self.url = url
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	979 self.code = code
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	980
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	981 def info(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	982 return self.headers
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	983
44817 3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	984 def getcode(self):
3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	985 return self.code
3f34d3b0350d #1178141: add addinfourl.code to get http status code from urllib. Georg Brandl <georg@python.org> parents: 44423 diff changeset	986
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	987 def geturl(self):
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	988 return self.url
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	989
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	990
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	991 # Utilities to parse URLs (most of these return None for missing parts):
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	992 # unwrap('<URL:type://host/path>') --> 'type://host/path'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	993 # splittype('type:opaquestring') --> 'type', 'opaquestring'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	994 # splithost('//host[:port]/path') --> 'host[:port]', '/path'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	995 # splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	996 # splitpasswd('user:passwd') -> 'user', 'passwd'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	997 # splitport('host:port') --> 'host', 'port'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	998 # splitquery('/path?query') --> '/path', 'query'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	999 # splittag('/path#tag') --> '/path', 'tag'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1000 # splitattr('/path;attr1=value1;attr2=value2;...') ->
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1001 # '/path', ['attr1=value1', 'attr2=value2', ...]
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1002 # splitvalue('attr=value') --> 'attr', 'value'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1003 # unquote('abc%20def') -> 'abc def'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1004 # quote('abc def') -> 'abc%20def')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1005
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1006 try:
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1007 unicode
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1008 except NameError:
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1009 def _is_unicode(x):
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1010 return 0
e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1011 else:
23540 bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1012 def _is_unicode(x):
bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1013 return isinstance(x, unicode)
bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1014
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1015 def toBytes(url):
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1016 """toBytes(u"URL") --> 'URL'."""
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1017 # Most URL schemes require ASCII. If that changes, the conversion
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1018 # can be relaxed
23540 bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1019 if _is_unicode(url):
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1020 try:
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1021 url = url.encode("ASCII")
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1022 except UnicodeError:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1023 raise UnicodeError("URL " + repr(url) +
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1024 " contains non-ASCII characters")
15468 21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1025 return url
21986cc14bb2 Convert Unicode strings to byte strings before passing them into specific Martin v. Löwis <martin@v.loewis.de> parents: 14981 diff changeset	1026
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1027 def unwrap(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1028 """unwrap('<URL:type://host/path>') --> 'type://host/path'."""
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1029 url = url.strip()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1030 if url[:1] == '<' and url[-1:] == '>':
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1031 url = url[1:-1].strip()
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1032 if url[:4] == 'URL:': url = url[4:].strip()
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1033 return url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1034
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1035 _typeprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1036 def splittype(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1037 """splittype('type:opaquestring') --> 'type', 'opaquestring'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1038 global _typeprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1039 if _typeprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1040 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1041 _typeprog = re.compile('^([^/:]+):')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1042
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1043 match = _typeprog.match(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1044 if match:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1045 scheme = match.group(1)
13287 4011acfe1433 Fix bug #314, reported by Craig Allen <cba@mediaone.net>: Fred Drake <fdrake@acm.org> parents: 12905 diff changeset	1046 return scheme.lower(), url[len(scheme) + 1:]
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1047 return None, url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1048
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1049 _hostprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1050 def splithost(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1051 """splithost('//host[:port]/path') --> 'host[:port]', '/path'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1052 global _hostprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1053 if _hostprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1054 import re
37228 8f7c03acaddd Bug #1457264: parse http://host?query correctly in urllib Georg Brandl <georg@python.org> parents: 36576 diff changeset	1055 _hostprog = re.compile('^//([^/?])(.)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1056
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1057 match = _hostprog.match(url)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1058 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1059 return None, url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1060
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1061 _userprog = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1062 def splituser(host):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1063 """splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1064 global _userprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1065 if _userprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1066 import re
24880 710bebe2182c Modify splituser() method to allow an @ in the userinfo field. Raymond Hettinger <python@rcn.com> parents: 24710 diff changeset	1067 _userprog = re.compile('^(.)@(.)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1068
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1069 match = _userprog.match(host)
14108 17d3128478c3 Patch from Paul Schreiber <paul@commerceflow.com>: Fred Drake <fdrake@acm.org> parents: 13718 diff changeset	1070 if match: return map(unquote, match.group(1, 2))
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1071 return None, host
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1072
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1073 _passwdprog = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1074 def splitpasswd(user):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1075 """splitpasswd('user:passwd') -> 'user', 'passwd'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1076 global _passwdprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1077 if _passwdprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1078 import re
52476 2ff89f0f1d19 Fix for bugs: Issue4675 and Issue4962. Senthil Kumaran <orsenthil@gmail.com> parents: 52265 diff changeset	1079 _passwdprog = re.compile('^([^:]):(.)$',re.S)
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1080
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1081 match = _passwdprog.match(user)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1082 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1083 return user, None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1084
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1085 # splittag('/path#tag') --> '/path', 'tag'
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1086 _portprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1087 def splitport(host):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1088 """splitport('host:port') --> 'host', 'port'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1089 global _portprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1090 if _portprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1091 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1092 _portprog = re.compile('^(.*):([0-9]+)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1093
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1094 match = _portprog.match(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1095 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1096 return host, None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1097
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1098 _nportprog = None
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	1099 def splitnport(host, defport=-1):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1100 """Split host and port, returning numeric port.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1101 Return given default port if no ':' found; defaults to -1.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1102 Return numerical port if a valid number are found after ':'.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1103 Return None if ':' but not a valid number."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1104 global _nportprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1105 if _nportprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1106 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1107 _nportprog = re.compile('^(.):(.)$')
7753 8ef067559d5a A few lines were indented using spaces instead of tabs -- fix them. Guido van Rossum <guido@python.org> parents: 7666 diff changeset	1108
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1109 match = _nportprog.match(host)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1110 if match:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1111 host, port = match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1112 try:
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1113 if not port: raise ValueError, "no digits"
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1114 nport = int(port)
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1115 except ValueError:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1116 nport = None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1117 return host, nport
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1118 return host, defport
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	1119
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1120 _queryprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1121 def splitquery(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1122 """splitquery('/path?query') --> '/path', 'query'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1123 global _queryprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1124 if _queryprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1125 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1126 _queryprog = re.compile('^(.)\?([^?])$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1127
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1128 match = _queryprog.match(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1129 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1130 return url, None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1131
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1132 _tagprog = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1133 def splittag(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1134 """splittag('/path#tag') --> '/path', 'tag'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1135 global _tagprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1136 if _tagprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1137 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1138 _tagprog = re.compile('^(.)#([^#])$')
7753 8ef067559d5a A few lines were indented using spaces instead of tabs -- fix them. Guido van Rossum <guido@python.org> parents: 7666 diff changeset	1139
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1140 match = _tagprog.match(url)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1141 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1142 return url, None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1143
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1144 def splitattr(url):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1145 """splitattr('/path;attr1=value1;attr2=value2;...') ->
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1146 '/path', ['attr1=value1', 'attr2=value2', ...]."""
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1147 words = url.split(';')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1148 return words[0], words[1:]
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1149
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1150 _valueprog = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1151 def splitvalue(attr):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1152 """splitvalue('attr=value') --> 'attr', 'value'."""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1153 global _valueprog
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1154 if _valueprog is None:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1155 import re
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1156 _valueprog = re.compile('^([^=])=(.)$')
6348 6a3b9e7d43c1 Use sys.exc_info() where needed. Guido van Rossum <guido@python.org> parents: 6268 diff changeset	1157
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1158 match = _valueprog.match(attr)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1159 if match: return match.group(1, 2)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1160 return attr, None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1161
35529 dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1162 _hextochr = dict(('%02x' % i, chr(i)) for i in range(256))
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1163 _hextochr.update(('%02X' % i, chr(i)) for i in range(256))
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1164
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1165 def unquote(s):
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1166 """unquote('abc%20def') -> 'abc def'."""
35529 dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1167 res = s.split('%')
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1168 for i in xrange(1, len(res)):
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1169 item = res[i]
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1170 try:
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1171 res[i] = _hextochr[item[:2]] + item[2:]
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1172 except KeyError:
dc8538845b0c Simplify and speed-up unquote(). Raymond Hettinger <python@rcn.com> parents: 35528 diff changeset	1173 res[i] = '%' + item
35723 ae539af4afc5 Teach unquote() to handle unicode inputs Raymond Hettinger <python@rcn.com> parents: 35531 diff changeset	1174 except UnicodeDecodeError:
ae539af4afc5 Teach unquote() to handle unicode inputs Raymond Hettinger <python@rcn.com> parents: 35531 diff changeset	1175 res[i] = unichr(int(item[:2], 16)) + item[2:]
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1176 return "".join(res)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1177
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	1178 def unquote_plus(s):
14118 44f4a84a80a8 * added doc strings to urlopen and unquote_plus Skip Montanaro <skip@pobox.com> parents: 14108 diff changeset	1179 """unquote('%7e/abc+def') -> '~/abc def'"""
31794 f1d44295ebf7 Replace sequential split/join calls on strings with a single replace call. Brett Cannon <bcannon@gmail.com> parents: 31788 diff changeset	1180 s = s.replace('+', ' ')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1181 return unquote(s)
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	1182
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1183 always_safe = ('ABCDEFGHIJKLMNOPQRSTUVWXYZ'
14297 eda449f54e44 fixes bug #111951 Jeremy Hylton <jeremy@alum.mit.edu> parents: 14181 diff changeset	1184 'abcdefghijklmnopqrstuvwxyz'
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1185 '0123456789' '_.-')
35527 5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1186 _safemaps = {}
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1187
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	1188 def quote(s, safe = '/'):
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1189 """quote('abc def') -> 'abc%20def'
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1190
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1191 Each part of a URL, e.g. the path info, the query, etc., has a
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1192 different set of reserved characters that must be quoted.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1193
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1194 RFC 2396 Uniform Resource Identifiers (URI): Generic Syntax lists
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1195 the following reserved characters.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1196
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1197 reserved = ";" \| "/" \| "?" \| ":" \| "@" \| "&" \| "=" \| "+" \|
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1198 "$" \| ","
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1199
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1200 Each of these characters is reserved in some component of a URL,
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1201 but not necessarily in all of them.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1202
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1203 By default, the quote function is intended for quoting the path
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1204 section of a URL. Thus, it will not encode '/'. This character
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1205 is reserved, but in typical usage the quote function is being
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1206 called on a path where the existing slash characters are used as
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1207 reserved characters.
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1208 """
35527 5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1209 cachekey = (safe, always_safe)
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1210 try:
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1211 safe_map = _safemaps[cachekey]
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1212 except KeyError:
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1213 safe += always_safe
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1214 safe_map = {}
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1215 for i in range(256):
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1216 c = chr(i)
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1217 safe_map[c] = (c in safe) and c or ('%%%02X' % i)
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1218 _safemaps[cachekey] = safe_map
5824130b7ac3 SF #1285086: urllib.quote is too slow Raymond Hettinger <python@rcn.com> parents: 35456 diff changeset	1219 res = map(safe_map.__getitem__, s)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1220 return ''.join(res)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1221
14551 28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1222 def quote_plus(s, safe = ''):
28df4f45967f Remove "," from the list of always_safe characters. It is a reserved Jeremy Hylton <jeremy@alum.mit.edu> parents: 14297 diff changeset	1223 """Quote the query fragment of a URL; replacing ' ' with '+'"""
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1224 if ' ' in s:
35531 39dbe3be327a Corrected version of 1.170 Raymond Hettinger <python@rcn.com> parents: 35530 diff changeset	1225 s = quote(s, safe + ' ')
39dbe3be327a Corrected version of 1.170 Raymond Hettinger <python@rcn.com> parents: 35530 diff changeset	1226 return s.replace(' ', '+')
39dbe3be327a Corrected version of 1.170 Raymond Hettinger <python@rcn.com> parents: 35530 diff changeset	1227 return quote(s, safe)
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	1228
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1229 def urlencode(query,doseq=0):
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1230 """Encode a sequence of two-element tuples or dictionary into a URL query string.
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1231
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1232 If any values in the query arg are sequences and doseq is true, each
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1233 sequence element is converted to a separate parameter.
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1234
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1235 If the query arg is a sequence of two-element tuples, the order of the
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1236 parameters in the output will match the order of parameters in the
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1237 input.
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1238 """
16535 084b3dc9b6ff Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16324 diff changeset	1239
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1240 if hasattr(query,"items"):
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1241 # mapping objects
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1242 query = query.items()
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1243 else:
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1244 # it's a bother at times that strings and string-like objects are
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1245 # sequences...
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1246 try:
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1247 # non-sequence items should not work with len()
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1248 # non-empty strings will fail this
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1249 if len(query) and not isinstance(query[0], tuple):
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1250 raise TypeError
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1251 # zero-length sequences of all types will get here and succeed,
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1252 # but that's a minor nit - since the original implementation
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1253 # allowed empty dicts that type of behavior probably should be
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1254 # preserved for consistency
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1255 except TypeError:
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1256 ty,va,tb = sys.exc_info()
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1257 raise TypeError, "not a valid non-string sequence or mapping object", tb
9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1258
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 11688 diff changeset	1259 l = []
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1260 if not doseq:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1261 # preserve old behavior
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1262 for k, v in query:
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1263 k = quote_plus(str(k))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1264 v = quote_plus(str(v))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1265 l.append(k + '=' + v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1266 else:
16324 9d572038206e allow first param urlencode to be a sequence of two-element tuples - in this Skip Montanaro <skip@pobox.com> parents: 16066 diff changeset	1267 for k, v in query:
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1268 k = quote_plus(str(k))
23645 e1e90b0f1f45 Remove uses of the string and types modules: Walter Dörwald <walter@livinglogic.de> parents: 23628 diff changeset	1269 if isinstance(v, str):
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1270 v = quote_plus(v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1271 l.append(k + '=' + v)
23540 bd58946aa521 Don't require Unicode support. Guido van Rossum <guido@python.org> parents: 23034 diff changeset	1272 elif _is_unicode(v):
16066 d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1273 # is there a reasonable way to convert to ASCII?
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1274 # encode generates a string, but "replace" or "ignore"
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1275 # lose information and "strict" can raise UnicodeError
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1276 v = quote_plus(v.encode("ASCII","replace"))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1277 l.append(k + '=' + v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1278 else:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1279 try:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1280 # is this a sufficient test for sequence-ness?
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1281 x = len(v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1282 except TypeError:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1283 # not a sequence
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1284 v = quote_plus(str(v))
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1285 l.append(k + '=' + v)
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1286 else:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1287 # loop over the sequence
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1288 for elt in v:
d9b69d780a16 modify urlencode so sequences in the dict are treated as multivalued Skip Montanaro <skip@pobox.com> parents: 16020 diff changeset	1289 l.append(k + '=' + quote_plus(str(elt)))
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1290 return '&'.join(l)
8700 95f324272f54 Speed up the implementation of quote(). Guido van Rossum <guido@python.org> parents: 8579 diff changeset	1291
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1292 # Proxy handling
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1293 def getproxies_environment():
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1294 """Return a dictionary of scheme -> proxy server URL mappings.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1295
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1296 Scan the environment for variables named <scheme>_proxy;
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1297 this seems to be the standard convention. If you need a
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1298 different way, you can pass a proxies dictionary to the
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1299 [Fancy]URLopener constructor.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1300
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1301 """
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1302 proxies = {}
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1303 for name, value in os.environ.items():
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1304 name = name.lower()
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1305 if value and name[-6:] == '_proxy':
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1306 proxies[name[:-6]] = value
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1307 return proxies
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1308
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1309 def proxy_bypass_environment(host):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1310 """Test if proxies should not be used for a particular host.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1311
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1312 Checks the environment for a variable named no_proxy, which should
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1313 be a list of DNS suffixes separated by commas, or '*' for all hosts.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1314 """
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1315 no_proxy = os.environ.get('no_proxy', '') or os.environ.get('NO_PROXY', '')
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1316 # '*' is special case for always bypass
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1317 if no_proxy == '*':
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1318 return 1
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1319 # strip port off host
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1320 hostonly, port = splitport(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1321 # check if the host ends with any of the DNS suffixes
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1322 for name in no_proxy.split(','):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1323 if name and (hostonly.endswith(name) or host.endswith(name)):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1324 return 1
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1325 # otherwise, don't bypass
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1326 return 0
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1327
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1328
32710 df88fadba2f5 Fix for #779167: use InternetConfig proxy settings on MacOSX (in addition Jack Jansen <jack.jansen@cwi.nl> parents: 31794 diff changeset	1329 if sys.platform == 'darwin':
47162 3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1330
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1331 def _CFSetup(sc):
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1332 from ctypes import c_int32, c_void_p, c_char_p, c_int
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1333 sc.CFStringCreateWithCString.argtypes = [ c_void_p, c_char_p, c_int32 ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1334 sc.CFStringCreateWithCString.restype = c_void_p
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1335 sc.SCDynamicStoreCopyProxies.argtypes = [ c_void_p ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1336 sc.SCDynamicStoreCopyProxies.restype = c_void_p
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1337 sc.CFDictionaryGetValue.argtypes = [ c_void_p, c_void_p ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1338 sc.CFDictionaryGetValue.restype = c_void_p
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1339 sc.CFStringGetLength.argtypes = [ c_void_p ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1340 sc.CFStringGetLength.restype = c_int32
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1341 sc.CFStringGetCString.argtypes = [ c_void_p, c_char_p, c_int32, c_int32 ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1342 sc.CFStringGetCString.restype = c_int32
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1343 sc.CFNumberGetValue.argtypes = [ c_void_p, c_int, c_void_p ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1344 sc.CFNumberGetValue.restype = c_int32
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1345 sc.CFRelease.argtypes = [ c_void_p ]
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1346 sc.CFRelease.restype = None
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1347
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1348 def _CStringFromCFString(sc, value):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1349 from ctypes import create_string_buffer
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1350 length = sc.CFStringGetLength(value) + 1
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1351 buff = create_string_buffer(length)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1352 sc.CFStringGetCString(value, buff, length, 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1353 return buff.value
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1354
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1355 def _CFNumberToInt32(sc, cfnum):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1356 from ctypes import byref, c_int
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1357 val = c_int()
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1358 kCFNumberSInt32Type = 3
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1359 sc.CFNumberGetValue(cfnum, kCFNumberSInt32Type, byref(val))
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1360 return val.value
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1361
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1362
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1363 def proxy_bypass_macosx_sysconf(host):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1364 """
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1365 Return True iff this host shouldn't be accessed using a proxy
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1366
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1367 This function uses the MacOSX framework SystemConfiguration
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1368 to fetch the proxy information.
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1369 """
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1370 from ctypes import cdll
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1371 from ctypes.util import find_library
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1372 import re
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1373 import socket
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1374 from fnmatch import fnmatch
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1375
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1376 def ip2num(ipAddr):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1377 parts = ipAddr.split('.')
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1378 parts = map(int, parts)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1379 if len(parts) != 4:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1380 parts = (parts + [0, 0, 0, 0])[:4]
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1381 return (parts[0] << 24) \| (parts[1] << 16) \| (parts[2] << 8) \| parts[3]
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1382
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1383 sc = cdll.LoadLibrary(find_library("SystemConfiguration"))
47162 3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1384 _CFSetup(sc)
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1385
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1386 hostIP = None
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1387
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1388 if not sc:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1389 return False
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1390
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1391 kSCPropNetProxiesExceptionsList = sc.CFStringCreateWithCString(0, "ExceptionsList", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1392 kSCPropNetProxiesExcludeSimpleHostnames = sc.CFStringCreateWithCString(0,
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1393 "ExcludeSimpleHostnames", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1394
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1395
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1396 proxyDict = sc.SCDynamicStoreCopyProxies(None)
47162 3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1397 if proxyDict is None:
3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1398 return False
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1399
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1400 try:
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1401 # Check for simple host names:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1402 if '.' not in host:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1403 exclude_simple = sc.CFDictionaryGetValue(proxyDict,
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1404 kSCPropNetProxiesExcludeSimpleHostnames)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1405 if exclude_simple and _CFNumberToInt32(sc, exclude_simple):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1406 return True
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1407
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1408
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1409 # Check the exceptions list:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1410 exceptions = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesExceptionsList)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1411 if exceptions:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1412 # Items in the list are strings like these: *.local, 169.254/16
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1413 for index in xrange(sc.CFArrayGetCount(exceptions)):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1414 value = sc.CFArrayGetValueAtIndex(exceptions, index)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1415 if not value: continue
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1416 value = _CStringFromCFString(sc, value)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1417
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1418 m = re.match(r"(\d+(?:\.\d+)*)(/\d+)?", value)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1419 if m is not None:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1420 if hostIP is None:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1421 hostIP = socket.gethostbyname(host)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1422 hostIP = ip2num(hostIP)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1423
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1424 base = ip2num(m.group(1))
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1425 mask = int(m.group(2)[1:])
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1426 mask = 32 - mask
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1427
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1428 if (hostIP >> mask) == (base >> mask):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1429 return True
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1430
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1431 elif fnmatch(host, value):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1432 return True
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1433
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1434 return False
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1435
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1436 finally:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1437 sc.CFRelease(kSCPropNetProxiesExceptionsList)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1438 sc.CFRelease(kSCPropNetProxiesExcludeSimpleHostnames)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1439
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1440
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1441
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1442 def getproxies_macosx_sysconf():
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1443 """Return a dictionary of scheme -> proxy server URL mappings.
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1444
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1445 This function uses the MacOSX framework SystemConfiguration
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1446 to fetch the proxy information.
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1447 """
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1448 from ctypes import cdll
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1449 from ctypes.util import find_library
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1450
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1451 sc = cdll.LoadLibrary(find_library("SystemConfiguration"))
47162 3071a337d921 MacOSX: ctypes annotation in implementation of getproxies_macosx_sysconf Ronald Oussoren <ronaldoussoren@mac.com> parents: 46943 diff changeset	1452 _CFSetup(sc)
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1453
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1454 if not sc:
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1455 return {}
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1456
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1457 kSCPropNetProxiesHTTPEnable = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1458 kSCPropNetProxiesHTTPProxy = sc.CFStringCreateWithCString(0, "HTTPProxy", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1459 kSCPropNetProxiesHTTPPort = sc.CFStringCreateWithCString(0, "HTTPPort", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1460
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1461 kSCPropNetProxiesHTTPSEnable = sc.CFStringCreateWithCString(0, "HTTPSEnable", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1462 kSCPropNetProxiesHTTPSProxy = sc.CFStringCreateWithCString(0, "HTTPSProxy", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1463 kSCPropNetProxiesHTTPSPort = sc.CFStringCreateWithCString(0, "HTTPSPort", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1464
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1465 kSCPropNetProxiesFTPEnable = sc.CFStringCreateWithCString(0, "FTPEnable", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1466 kSCPropNetProxiesFTPPassive = sc.CFStringCreateWithCString(0, "FTPPassive", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1467 kSCPropNetProxiesFTPPort = sc.CFStringCreateWithCString(0, "FTPPort", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1468 kSCPropNetProxiesFTPProxy = sc.CFStringCreateWithCString(0, "FTPProxy", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1469
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1470 kSCPropNetProxiesGopherEnable = sc.CFStringCreateWithCString(0, "GopherEnable", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1471 kSCPropNetProxiesGopherPort = sc.CFStringCreateWithCString(0, "GopherPort", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1472 kSCPropNetProxiesGopherProxy = sc.CFStringCreateWithCString(0, "GopherProxy", 0)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1473
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1474 proxies = {}
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1475 proxyDict = sc.SCDynamicStoreCopyProxies(None)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1476
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1477 try:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1478 # HTTP:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1479 enabled = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesHTTPEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1480 if enabled and _CFNumberToInt32(sc, enabled):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1481 proxy = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesHTTPProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1482 port = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesHTTPPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1483
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1484 if proxy:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1485 proxy = _CStringFromCFString(sc, proxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1486 if port:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1487 port = _CFNumberToInt32(sc, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1488 proxies["http"] = "http://%s:%i" % (proxy, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1489 else:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1490 proxies["http"] = "http://%s" % (proxy, )
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1491
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1492 # HTTPS:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1493 enabled = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesHTTPSEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1494 if enabled and _CFNumberToInt32(sc, enabled):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1495 proxy = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesHTTPSProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1496 port = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesHTTPSPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1497
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1498 if proxy:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1499 proxy = _CStringFromCFString(sc, proxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1500 if port:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1501 port = _CFNumberToInt32(sc, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1502 proxies["https"] = "http://%s:%i" % (proxy, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1503 else:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1504 proxies["https"] = "http://%s" % (proxy, )
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1505
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1506 # FTP:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1507 enabled = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesFTPEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1508 if enabled and _CFNumberToInt32(sc, enabled):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1509 proxy = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesFTPProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1510 port = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesFTPPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1511
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1512 if proxy:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1513 proxy = _CStringFromCFString(sc, proxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1514 if port:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1515 port = _CFNumberToInt32(sc, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1516 proxies["ftp"] = "http://%s:%i" % (proxy, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1517 else:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1518 proxies["ftp"] = "http://%s" % (proxy, )
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1519
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1520 # Gopher:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1521 enabled = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesGopherEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1522 if enabled and _CFNumberToInt32(sc, enabled):
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1523 proxy = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesGopherProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1524 port = sc.CFDictionaryGetValue(proxyDict, kSCPropNetProxiesGopherPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1525
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1526 if proxy:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1527 proxy = _CStringFromCFString(sc, proxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1528 if port:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1529 port = _CFNumberToInt32(sc, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1530 proxies["gopher"] = "http://%s:%i" % (proxy, port)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1531 else:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1532 proxies["gopher"] = "http://%s" % (proxy, )
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1533 finally:
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1534 sc.CFRelease(proxyDict)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1535
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1536 sc.CFRelease(kSCPropNetProxiesHTTPEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1537 sc.CFRelease(kSCPropNetProxiesHTTPProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1538 sc.CFRelease(kSCPropNetProxiesHTTPPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1539 sc.CFRelease(kSCPropNetProxiesFTPEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1540 sc.CFRelease(kSCPropNetProxiesFTPPassive)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1541 sc.CFRelease(kSCPropNetProxiesFTPPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1542 sc.CFRelease(kSCPropNetProxiesFTPProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1543 sc.CFRelease(kSCPropNetProxiesGopherEnable)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1544 sc.CFRelease(kSCPropNetProxiesGopherPort)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1545 sc.CFRelease(kSCPropNetProxiesGopherProxy)
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1546
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1547 return proxies
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1548
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1549
089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1550
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1551 def proxy_bypass(host):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1552 if getproxies_environment():
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1553 return proxy_bypass_environment(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1554 else:
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1555 return proxy_bypass_macosx_sysconf(host)
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1556
32710 df88fadba2f5 Fix for #779167: use InternetConfig proxy settings on MacOSX (in addition Jack Jansen <jack.jansen@cwi.nl> parents: 31794 diff changeset	1557 def getproxies():
46943 089c8d0df774 MacOSX: remove dependency on Carbon package for urllib Ronald Oussoren <ronaldoussoren@mac.com> parents: 44819 diff changeset	1558 return getproxies_environment() or getproxies_macosx_sysconf()
32736 c3a096bfbd6b Whitespace normalization, via reindent.py. Tim Peters <tim.peters@gmail.com> parents: 32710 diff changeset	1559
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1560 elif os.name == 'nt':
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1561 def getproxies_registry():
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1562 """Return a dictionary of scheme -> proxy server URL mappings.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1563
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1564 Win32 uses the registry to store proxies.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1565
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1566 """
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1567 proxies = {}
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1568 try:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1569 import _winreg
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1570 except ImportError:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1571 # Std module, so should be around - but you never know!
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1572 return proxies
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1573 try:
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1574 internetSettings = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER,
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1575 r'Software\Microsoft\Windows\CurrentVersion\Internet Settings')
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1576 proxyEnable = _winreg.QueryValueEx(internetSettings,
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1577 'ProxyEnable')[0]
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1578 if proxyEnable:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1579 # Returned as Unicode but problems if not converted to ASCII
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1580 proxyServer = str(_winreg.QueryValueEx(internetSettings,
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1581 'ProxyServer')[0])
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1582 if '=' in proxyServer:
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1583 # Per-protocol settings
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1584 for p in proxyServer.split(';'):
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1585 protocol, address = p.split('=', 1)
22717 0bdd413a3416 Hopeful fix for SF bug 503031: urllib.py: open_http() host problem. Guido van Rossum <guido@python.org> parents: 22588 diff changeset	1586 # See if address has a type:// prefix
22768 3a5065005fcf Fix for a bug in the fix for SF bug 503031. This time the OP verified Guido van Rossum <guido@python.org> parents: 22717 diff changeset	1587 import re
3a5065005fcf Fix for a bug in the fix for SF bug 503031. This time the OP verified Guido van Rossum <guido@python.org> parents: 22717 diff changeset	1588 if not re.match('^([^/:]+)://', address):
22717 0bdd413a3416 Hopeful fix for SF bug 503031: urllib.py: open_http() host problem. Guido van Rossum <guido@python.org> parents: 22588 diff changeset	1589 address = '%s://%s' % (protocol, address)
0bdd413a3416 Hopeful fix for SF bug 503031: urllib.py: open_http() host problem. Guido van Rossum <guido@python.org> parents: 22588 diff changeset	1590 proxies[protocol] = address
14803 29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1591 else:
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1592 # Use one setting for all protocols
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1593 if proxyServer[:5] == 'http:':
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1594 proxies['http'] = proxyServer
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1595 else:
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1596 proxies['http'] = 'http://%s' % proxyServer
29902f7748fa - Improved handling of win32 proxy settings (addresses bug #114256). Fredrik Lundh <fredrik@pythonware.com> parents: 14551 diff changeset	1597 proxies['ftp'] = 'ftp://%s' % proxyServer
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1598 internetSettings.Close()
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1599 except (WindowsError, ValueError, TypeError):
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1600 # Either registry key not found etc, or the value in an
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1601 # unexpected format.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1602 # proxies already set up to be empty so nothing to do
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1603 pass
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1604 return proxies
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1605
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1606 def getproxies():
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1607 """Return a dictionary of scheme -> proxy server URL mappings.
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1608
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1609 Returns settings gathered from the environment, if specified,
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1610 or the registry.
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1611
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1612 """
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1613 return getproxies_environment() or getproxies_registry()
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1614
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1615 def proxy_bypass_registry(host):
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1616 try:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1617 import _winreg
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1618 import re
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1619 except ImportError:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1620 # Std modules, so should be around - but you never know!
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1621 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1622 try:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1623 internetSettings = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER,
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1624 r'Software\Microsoft\Windows\CurrentVersion\Internet Settings')
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1625 proxyEnable = _winreg.QueryValueEx(internetSettings,
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1626 'ProxyEnable')[0]
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1627 proxyOverride = str(_winreg.QueryValueEx(internetSettings,
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1628 'ProxyOverride')[0])
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1629 # ^^^^ Returned as Unicode but problems if not converted to ASCII
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1630 except WindowsError:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1631 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1632 if not proxyEnable or not proxyOverride:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1633 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1634 # try to make a host list from name and IP address.
36576 33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1635 rawHost, port = splitport(host)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1636 host = [rawHost]
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1637 try:
36576 33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1638 addr = socket.gethostbyname(rawHost)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1639 if addr != rawHost:
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1640 host.append(addr)
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1641 except socket.error:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1642 pass
36576 33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1643 try:
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1644 fqdn = socket.getfqdn(rawHost)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1645 if fqdn != rawHost:
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1646 host.append(fqdn)
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1647 except socket.error:
33adc5dd2bf2 Patch #1426648: urllib proxy_bypass broken Georg Brandl <georg@python.org> parents: 36402 diff changeset	1648 pass
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1649 # make a check value list from the registry entry: replace the
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1650 # '<local>' string by the localhost entry and the corresponding
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1651 # canonical entry.
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1652 proxyOverride = proxyOverride.split(';')
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1653 # now check if we match one of the registry values.
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1654 for test in proxyOverride:
53636 29656a03cce2 Fix for Issue1648102, based on the MSDN spec: If this parameter specifies the Senthil Kumaran <orsenthil@gmail.com> parents: 53297 diff changeset	1655 if test == '<local>':
29656a03cce2 Fix for Issue1648102, based on the MSDN spec: If this parameter specifies the Senthil Kumaran <orsenthil@gmail.com> parents: 53297 diff changeset	1656 if '.' not in rawHost:
29656a03cce2 Fix for Issue1648102, based on the MSDN spec: If this parameter specifies the Senthil Kumaran <orsenthil@gmail.com> parents: 53297 diff changeset	1657 return 1
19119 600a9a2b7a18 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 19104 diff changeset	1658 test = test.replace(".", r"\.") # mask dots
600a9a2b7a18 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 19104 diff changeset	1659 test = test.replace("", r".") # change glob sequence
600a9a2b7a18 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 19104 diff changeset	1660 test = test.replace("?", r".") # change glob char
19104 875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1661 for val in host:
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1662 # print "%s <--> %s" %( test, val )
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1663 if re.match(test, val, re.I):
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1664 return 1
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1665 return 0
875ac8e93542 SF patch #403640: incomplete proxy handling in URLLIB Tim Peters <tim.peters@gmail.com> parents: 19103 diff changeset	1666
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1667 def proxy_bypass(host):
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1668 """Return a dictionary of scheme -> proxy server URL mappings.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1669
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1670 Returns settings gathered from the environment, if specified,
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1671 or the registry.
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1672
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1673 """
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1674 if getproxies_environment():
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1675 return proxy_bypass_environment(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1676 else:
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1677 return proxy_bypass_registry(host)
e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1678
13718 17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1679 else:
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1680 # By default use environment variables
17293f12e884 Patch #100873 - Use registry to find proxies for urllib on Win32 Mark Hammond <mhammond@skippinet.com.au> parents: 13620 diff changeset	1681 getproxies = getproxies_environment
44818 e36469e6d8b6 #856047: respect the ``no_proxy`` env var when checking for proxies Georg Brandl <georg@python.org> parents: 44817 diff changeset	1682 proxy_bypass = proxy_bypass_environment
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	1683
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1684 # Test and time quote() and unquote()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1685 def test1():
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1686 s = ''
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1687 for i in range(256): s = s + chr(i)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1688 s = s*4
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1689 t0 = time.time()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1690 qs = quote(s)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1691 uqs = unquote(qs)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1692 t1 = time.time()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1693 if uqs != s:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1694 print 'Wrong!'
31515 71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	1695 print repr(s)
71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	1696 print repr(qs)
71a294bd7950 Replace backticks with repr() or "%r" Walter Dörwald <walter@livinglogic.de> parents: 30232 diff changeset	1697 print repr(uqs)
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1698 print round(t1 - t0, 3), 'sec'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1699
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1700
9148 e60055216b8f Changes by Eric Raymond: Guido van Rossum <guido@python.org> parents: 8848 diff changeset	1701 def reporthook(blocknum, blocksize, totalsize):
e60055216b8f Changes by Eric Raymond: Guido van Rossum <guido@python.org> parents: 8848 diff changeset	1702 # Report during remote transfers
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1703 print "Block number: %d, Block size: %d, Total size: %d" % (
afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1704 blocknum, blocksize, totalsize)
9148 e60055216b8f Changes by Eric Raymond: Guido van Rossum <guido@python.org> parents: 8848 diff changeset	1705
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1706 # Test program
8547 d84f1209cba5 Rewrite the (test) main program so that when used as a script, it can Guido van Rossum <guido@python.org> parents: 8493 diff changeset	1707 def test(args=[]):
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1708 if not args:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1709 args = [
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1710 '/etc/passwd',
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1711 'file:/etc/passwd',
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1712 'file://localhost/etc/passwd',
41116 9a0ccdbbc568 Patch #1678662: ftp.python.org does not exist. So the testcode in urllib.py must use a more Collin Winter <collinw@gmail.com> parents: 40689 diff changeset	1713 'ftp://ftp.gnu.org/pub/README',
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1714 'http://www.python.org/index.html',
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1715 ]
11527 28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	1716 if hasattr(URLopener, "open_https"):
28b8585354de OpenSSL support. This is based on patches for a version of SSLeay by Guido van Rossum <guido@python.org> parents: 11201 diff changeset	1717 args.append('https://synergy.as.cmu.edu/~geek/')
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1718 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1719 for url in args:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1720 print '-'10, url, '-'10
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1721 fn, h = urlretrieve(url, None, reporthook)
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1722 print fn
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1723 if h:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1724 print '======'
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1725 for k in h.keys(): print k + ':', h[k]
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1726 print '======'
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1727 fp = open(fn, 'rb')
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1728 data = fp.read()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1729 del fp
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1730 if '\r' in data:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1731 table = string.maketrans("", "")
15581 afb45b7fa252 Get rid of string functions, except maketrans() (which is not Guido van Rossum <guido@python.org> parents: 15468 diff changeset	1732 data = data.translate(table, "\r")
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1733 print data
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1734 fn, h = None, None
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1735 print '-'*40
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1736 finally:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1737 urlcleanup()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1738
8547 d84f1209cba5 Rewrite the (test) main program so that when used as a script, it can Guido van Rossum <guido@python.org> parents: 8493 diff changeset	1739 def main():
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1740 import getopt, sys
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1741 try:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1742 opts, args = getopt.getopt(sys.argv[1:], "th")
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1743 except getopt.error, msg:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1744 print msg
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1745 print "Use -h for help"
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1746 return
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1747 t = 0
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1748 for o, a in opts:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1749 if o == '-t':
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1750 t = t + 1
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1751 if o == '-h':
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1752 print "Usage: python urllib.py [-t] [url ...]"
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1753 print "-t runs self-test;",
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1754 print "otherwise, contents of urls are printed"
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1755 return
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1756 if t:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1757 if t > 1:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1758 test1()
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1759 test(args)
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1760 else:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1761 if not args:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1762 print "Use -h for help"
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1763 for url in args:
e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1764 print urlopen(url).read(),
8547 d84f1209cba5 Rewrite the (test) main program so that when used as a script, it can Guido van Rossum <guido@python.org> parents: 8493 diff changeset	1765
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1766 # Run test program when run as a script
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1767 if __name__ == '__main__':
10307 e7c09a62bf1f change indentation from 8 spaces to 4 spaces Jeremy Hylton <jeremy@alum.mit.edu> parents: 10306 diff changeset	1768 main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/urllib.py @ 54294:dba767bc3155 legacy-trunk