cpython-lru_cache-weakref: Lib/urllib.py annotate

annotate Lib/urllib.py @ 5817:48b94de85bf3 legacy-trunk

Extensive changes by AMK.

author	Guido van Rossum <guido@python.org>
date	Thu, 17 Jul 1997 16:29:42 +0000
parents	e0da3d89d4b6
children	c20b6980a870

rev	line source
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	1 # Open an arbitrary URL
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	2 #
5089 91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	3 # See the following document for more info on URLs:
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	4 # "Names and Addresses, URIs, URLs, URNs, URCs", at
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	5 # http://www.w3.org/pub/WWW/Addressing/Overview.html
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	6 #
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	7 # See also the HTTP spec (from which the error codes are derived):
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	8 # "HTTP - Hypertext Transfer Protocol", at
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	9 # http://www.w3.org/pub/WWW/Protocols/
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	10 #
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	11 # Related standards and specs:
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	12 # - RFC1808: the "relative URL" spec. (authoritative status)
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	13 # - RFC1738 - the "URL standard". (authoritative status)
91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	14 # - RFC1630 - the "URI spec". (informational status)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	15 #
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	16 # The object returned by URLopener().open(file) will differ per
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	17 # protocol. All you know is that is has methods read(), readline(),
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	18 # readlines(), fileno(), close() and info(). The read*(), fileno()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	19 # and close() methods work like those of open files.
5089 91f86306b07b Put a new, more useful, set of references in the leading comment. Guido van Rossum <guido@python.org> parents: 5072 diff changeset	20 # The info() method returns a mimetools.Message object which can be
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	21 # used to query various info about the object, if available.
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	22 # (mimetools.Message objects are queried with the getheader() method.)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	23
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	24 import string
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	25 import socket
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	26 import regex
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	27 import os
4558 df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	28 import sys
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	29
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	30
5727 e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	31 __version__ = '1.7'
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	32
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	33 MAXFTPCACHE = 10 # Trim the ftp cache beyond this size
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	34
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	35 # Helper for non-unix systems
9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	36 if os.name == 'mac':
3471 aa8adfdd3aa1 move mac url2path conversion to separate module Guido van Rossum <guido@python.org> parents: 3461 diff changeset	37 from macurl2path import url2pathname, pathname2url
3724 26442b5f9cfd add nturl2path Guido van Rossum <guido@python.org> parents: 3687 diff changeset	38 elif os.name == 'nt':
26442b5f9cfd add nturl2path Guido van Rossum <guido@python.org> parents: 3687 diff changeset	39 from nturl2path import url2pathname, pathname2url
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	40 else:
3303 d4d5c9fa7859 Added pathname2url and url2pathname methods (only correct for unix and Jack Jansen <jack.jansen@cwi.nl> parents: 3298 diff changeset	41 def url2pathname(pathname):
3298 9a4081440783 Fixed local file access for macintosh Jack Jansen <jack.jansen@cwi.nl> parents: 3244 diff changeset	42 return pathname
3303 d4d5c9fa7859 Added pathname2url and url2pathname methods (only correct for unix and Jack Jansen <jack.jansen@cwi.nl> parents: 3298 diff changeset	43 def pathname2url(pathname):
d4d5c9fa7859 Added pathname2url and url2pathname methods (only correct for unix and Jack Jansen <jack.jansen@cwi.nl> parents: 3298 diff changeset	44 return pathname
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	45
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	46 # This really consists of two pieces:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	47 # (1) a class which handles opening of all sorts of URLs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	48 # (plus assorted utilities etc.)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	49 # (2) a set of functions for parsing URLs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	50 # XXX Should these be separated out into different modules?
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	51
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	52
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	53 # Shortcut for basic usage
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	54 _urlopener = None
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	55 def urlopen(url, data=None):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	56 global _urlopener
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	57 if not _urlopener:
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	58 _urlopener = FancyURLopener()
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	59 if data is None:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	60 return _urlopener.open(url)
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	61 else:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	62 return _urlopener.open(url, data)
3660 058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	63 def urlretrieve(url, filename=None):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	64 global _urlopener
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	65 if not _urlopener:
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	66 _urlopener = FancyURLopener()
3660 058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	67 if filename:
058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	68 return _urlopener.retrieve(url, filename)
058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	69 else:
058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	70 return _urlopener.retrieve(url)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	71 def urlcleanup():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	72 if _urlopener:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	73 _urlopener.cleanup()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	74
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	75
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	76 # Class to open URLs.
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	77 # This is a class rather than just a subroutine because we may need
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	78 # more than one set of global protocol-specific options.
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	79 # Note -- this is a base class for those who don't want the
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	80 # automatic handling of errors type 302 (relocated) and 401
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	81 # (authorization needed).
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	82 ftpcache = {}
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	83 class URLopener:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	84
4566 6403c94de50d Added safeguard against failure in __del__. Guido van Rossum <guido@python.org> parents: 4558 diff changeset	85 tempcache = None # So close() in __del__() won't fail
6403c94de50d Added safeguard against failure in __del__. Guido van Rossum <guido@python.org> parents: 4558 diff changeset	86
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	87 # Constructor
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	88 def __init__(self, proxies=None):
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	89 if proxies is None:
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	90 proxies = getproxies()
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	91 self.proxies = proxies
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	92 server_version = "Python-urllib/%s" % __version__
af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	93 self.addheaders = [('User-agent', server_version)]
1810 d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	94 self.tempcache = None
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	95 # Undocumented feature: if you assign {} to tempcache,
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	96 # it is used to cache files retrieved with
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	97 # self.retrieve(). This is not enabled by default
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	98 # since it does not work for changing documents (and I
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	99 # haven't got the logic to check expiration headers
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	100 # yet).
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	101 self.ftpcache = ftpcache
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	102 # Undocumented feature: you can use a different
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	103 # ftp cache by assigning to the .ftpcache member;
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	104 # in case you want logically independent URL openers
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	105
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	106 def __del__(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	107 self.close()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	108
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	109 def close(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	110 self.cleanup()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	111
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	112 def cleanup(self):
1810 d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	113 if self.tempcache:
5072 d2483c27742c Move the 'import os' in URLopener.cleanup() to inside the block Guido van Rossum <guido@python.org> parents: 4715 diff changeset	114 import os
1810 d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	115 for url in self.tempcache.keys():
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	116 try:
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	117 os.unlink(self.tempcache[url][0])
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	118 except os.error:
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	119 pass
d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	120 del self.tempcache[url]
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	121
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	122 # Add a header to be used by the HTTP interface only
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	123 # e.g. u.addheader('Accept', 'sound/basic')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	124 def addheader(self, *args):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	125 self.addheaders.append(args)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	126
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	127 # External interface
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	128 # Use URLopener().open(file) instead of open(file, 'r')
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	129 def open(self, fullurl, data=None):
3028 7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	130 fullurl = unwrap(fullurl)
7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	131 type, url = splittype(fullurl)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	132 if not type: type = 'file'
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	133 self.openedurl = '%s:%s' % (type, url)
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	134 if self.proxies.has_key(type):
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	135 proxy = self.proxies[type]
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	136 type, proxy = splittype(proxy)
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	137 host, selector = splithost(proxy)
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	138 url = (host, fullurl) # Signal special case to open_*()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	139 name = 'open_' + type
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	140 if '-' in name:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	141 import regsub
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	142 name = regsub.gsub('-', '_', name)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	143 if not hasattr(self, name):
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	144 if data is None:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	145 return self.open_unknown(fullurl)
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	146 else:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	147 return self.open_unknown(fullurl, data)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	148 try:
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	149 if data is None:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	150 return getattr(self, name)(url)
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	151 else:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	152 return getattr(self, name)(url, data)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	153 except socket.error, msg:
4558 df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	154 raise IOError, ('socket error', msg), sys.exc_traceback
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	155
3028 7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	156 # Overridable interface to open unknown URL type
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	157 def open_unknown(self, fullurl, data=None):
3028 7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	158 type, url = splittype(fullurl)
7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	159 raise IOError, ('url error', 'unknown url type', type)
7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	160
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	161 # External interface
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	162 # retrieve(url) returns (filename, None) for a local object
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	163 # or (tempfilename, headers) for a remote object
3660 058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	164 def retrieve(self, url, filename=None):
1810 d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	165 if self.tempcache and self.tempcache.has_key(url):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	166 return self.tempcache[url]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	167 url1 = unwrap(url)
4440 fce296dbb03e Fix another case where... Guido van Rossum <guido@python.org> parents: 4395 diff changeset	168 self.openedurl = url1
1810 d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	169 if self.tempcache and self.tempcache.has_key(url1):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	170 self.tempcache[url] = self.tempcache[url1]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	171 return self.tempcache[url1]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	172 type, url1 = splittype(url1)
3660 058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	173 if not filename and (not type or type == 'file'):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	174 try:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	175 fp = self.open_local_file(url1)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	176 del fp
3303 d4d5c9fa7859 Added pathname2url and url2pathname methods (only correct for unix and Jack Jansen <jack.jansen@cwi.nl> parents: 3298 diff changeset	177 return url2pathname(splithost(url1)[1]), None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	178 except IOError, msg:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	179 pass
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	180 fp = self.open(url)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	181 headers = fp.info()
3660 058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	182 if not filename:
058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	183 import tempfile
058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	184 filename = tempfile.mktemp()
058d81d1d56d Support optional filename argument for retrieve() and urlretrieve(), Guido van Rossum <guido@python.org> parents: 3471 diff changeset	185 result = filename, headers
1810 d892ac4b4fb9 * Lib/linecache.py: don't crash on empty filename Guido van Rossum <guido@python.org> parents: 1778 diff changeset	186 if self.tempcache is not None:
1899 a6776a788f74 Fix bug if tmpcache is None Guido van Rossum <guido@python.org> parents: 1828 diff changeset	187 self.tempcache[url] = result
5306 e614bcd34e34 Open files in binary mode. Guido van Rossum <guido@python.org> parents: 5241 diff changeset	188 tfp = open(filename, 'wb')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	189 bs = 1024*8
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	190 block = fp.read(bs)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	191 while block:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	192 tfp.write(block)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	193 block = fp.read(bs)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	194 del fp
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	195 del tfp
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	196 return result
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	197
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	198 # Each method named open_<type> knows how to open that type of URL
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	199
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	200 # Use HTTP protocol
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	201 def open_http(self, url, data=None):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	202 import httplib
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	203 if type(url) is type(""):
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	204 host, selector = splithost(url)
4153 cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	205 user_passwd, host = splituser(host)
5709 671832470aa1 Add Host: header to URL request. Guido van Rossum <guido@python.org> parents: 5653 diff changeset	206 realhost = host
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	207 else:
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	208 host, selector = url
4153 cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	209 urltype, rest = splittype(selector)
5241 b89f52821c52 Fix two small bugs with proxies. Guido van Rossum <guido@python.org> parents: 5089 diff changeset	210 user_passwd = None
5709 671832470aa1 Add Host: header to URL request. Guido van Rossum <guido@python.org> parents: 5653 diff changeset	211 if string.lower(urltype) != 'http':
671832470aa1 Add Host: header to URL request. Guido van Rossum <guido@python.org> parents: 5653 diff changeset	212 realhost = None
671832470aa1 Add Host: header to URL request. Guido van Rossum <guido@python.org> parents: 5653 diff changeset	213 else:
4153 cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	214 realhost, rest = splithost(rest)
cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	215 user_passwd, realhost = splituser(realhost)
cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	216 if user_passwd:
cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	217 selector = "%s://%s%s" % (urltype,
cc06e70127fc Bump exposed __version__ to 1.4. Guido van Rossum <guido@python.org> parents: 4146 diff changeset	218 realhost, rest)
5241 b89f52821c52 Fix two small bugs with proxies. Guido van Rossum <guido@python.org> parents: 5089 diff changeset	219 #print "proxy via http:", host, selector
1590 ac2e5cc29949 Added tests for missing host to open_http and open_gopher Guido van Rossum <guido@python.org> parents: 1578 diff changeset	220 if not host: raise IOError, ('http error', 'no host given')
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	221 if user_passwd:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	222 import base64
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	223 auth = string.strip(base64.encodestring(user_passwd))
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	224 else:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	225 auth = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	226 h = httplib.HTTP(host)
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	227 if data is not None:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	228 h.putrequest('POST', selector)
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	229 h.putheader('Content-type',
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	230 'application/x-www-form-urlencoded')
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	231 h.putheader('Content-length', '%d' % len(data))
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	232 else:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	233 h.putrequest('GET', selector)
4532 30ceca3f105b Fix the way the Authorization header is sent (how could this have worked?). Guido van Rossum <guido@python.org> parents: 4440 diff changeset	234 if auth: h.putheader('Authorization', 'Basic %s' % auth)
5709 671832470aa1 Add Host: header to URL request. Guido van Rossum <guido@python.org> parents: 5653 diff changeset	235 if realhost: h.putheader('Host', realhost)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	236 for args in self.addheaders: apply(h.putheader, args)
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	237 h.endheaders()
4659 e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	238 if data is not None:
e97528482bc6 Add optional data argument to urlopen() and open_http(), to trigger POST. Guido van Rossum <guido@python.org> parents: 4566 diff changeset	239 h.send(data + '\r\n')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	240 errcode, errmsg, headers = h.getreply()
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	241 fp = h.getfile()
af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	242 if errcode == 200:
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	243 return addinfourl(fp, headers, self.openedurl)
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	244 else:
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	245 return self.http_error(url,
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	246 fp, errcode, errmsg, headers)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	247
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	248 # Handle http errors.
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	249 # Derived class can override this, or provide specific handlers
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	250 # named http_error_DDD where DDD is the 3-digit error code
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	251 def http_error(self, url, fp, errcode, errmsg, headers):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	252 # First check if there's a specific handler for this error
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	253 name = 'http_error_%d' % errcode
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	254 if hasattr(self, name):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	255 method = getattr(self, name)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	256 result = method(url, fp, errcode, errmsg, headers)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	257 if result: return result
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	258 return self.http_error_default(
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	259 url, fp, errcode, errmsg, headers)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	260
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	261 # Default http error handler: close the connection and raises IOError
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	262 def http_error_default(self, url, fp, errcode, errmsg, headers):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	263 void = fp.read()
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	264 fp.close()
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	265 raise IOError, ('http error', errcode, errmsg, headers)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	266
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	267 # Use Gopher protocol
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	268 def open_gopher(self, url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	269 import gopherlib
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	270 host, selector = splithost(url)
1590 ac2e5cc29949 Added tests for missing host to open_http and open_gopher Guido van Rossum <guido@python.org> parents: 1578 diff changeset	271 if not host: raise IOError, ('gopher error', 'no host given')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	272 type, selector = splitgophertype(selector)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	273 selector, query = splitquery(selector)
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	274 selector = unquote(selector)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	275 if query:
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	276 query = unquote(query)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	277 fp = gopherlib.send_query(selector, query, host)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	278 else:
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	279 fp = gopherlib.send_selector(selector, host)
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	280 return addinfourl(fp, noheaders(), self.openedurl)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	281
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	282 # Use local file or FTP depending on form of URL
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	283 def open_file(self, url):
3028 7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	284 if url[:2] == '//':
7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	285 return self.open_ftp(url)
7cd1da4064d7 support overriding how to open unknown url types Guido van Rossum <guido@python.org> parents: 2996 diff changeset	286 else:
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	287 return self.open_local_file(url)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	288
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	289 # Use local file
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	290 def open_local_file(self, url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	291 host, file = splithost(url)
4395 48d1f5beb16b Fix some cases where self.openedurl wasn't set. Guido van Rossum <guido@python.org> parents: 4257 diff changeset	292 if not host:
5726 57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	293 return addinfourl(
57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	294 open(url2pathname(file), 'rb'),
57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	295 noheaders(), 'file:'+file)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	296 host, port = splitport(host)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	297 if not port and socket.gethostbyname(host) in (
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	298 localhost(), thishost()):
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	299 file = unquote(file)
5726 57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	300 return addinfourl(
57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	301 open(url2pathname(file), 'rb'),
57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	302 noheaders(), 'file:'+file)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	303 raise IOError, ('local file error', 'not on local host')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	304
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	305 # Use FTP protocol
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	306 def open_ftp(self, url):
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	307 host, path = splithost(url)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	308 if not host: raise IOError, ('ftp error', 'no host given')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	309 host, port = splitport(host)
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	310 user, host = splituser(host)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	311 if user: user, passwd = splitpasswd(user)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	312 else: passwd = None
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	313 host = socket.gethostbyname(host)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	314 if not port:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	315 import ftplib
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	316 port = ftplib.FTP_PORT
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	317 path, attrs = splitattr(path)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	318 dirs = string.splitfields(path, '/')
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	319 dirs, file = dirs[:-1], dirs[-1]
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	320 if dirs and not dirs[0]: dirs = dirs[1:]
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	321 key = (user, host, port, string.joinfields(dirs, '/'))
5727 e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	322 if len(self.ftpcache) > MAXFTPCACHE:
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	323 # Prune the cache, rather arbitrarily
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	324 for k in self.ftpcache.keys():
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	325 if k != key:
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	326 v = self.ftpcache[k]
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	327 del self.ftpcache[k]
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	328 v.close()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	329 try:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	330 if not self.ftpcache.has_key(key):
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	331 self.ftpcache[key] = \
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	332 ftpwrapper(user, passwd,
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	333 host, port, dirs)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	334 if not file: type = 'D'
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	335 else: type = 'I'
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	336 for attr in attrs:
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	337 attr, value = splitvalue(attr)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	338 if string.lower(attr) == 'type' and \
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	339 value in ('a', 'A', 'i', 'I', 'd', 'D'):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	340 type = string.upper(value)
5726 57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	341 return addinfourl(
57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	342 self.ftpcache[key].retrfile(file, type),
57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	343 noheaders(), self.openedurl)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	344 except ftperrors(), msg:
4558 df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	345 raise IOError, ('ftp error', msg), sys.exc_traceback
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	346
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	347
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	348 # Derived class with handlers for errors we can handle (perhaps)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	349 class FancyURLopener(URLopener):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	350
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	351 def __init__(self, *args):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	352 apply(URLopener.__init__, (self,) + args)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	353 self.auth_cache = {}
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	354
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	355 # Default error handling -- don't raise an exception
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	356 def http_error_default(self, url, fp, errcode, errmsg, headers):
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	357 return addinfourl(fp, headers, self.openedurl)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	358
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	359 # Error 302 -- relocated (temporarily)
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	360 def http_error_302(self, url, fp, errcode, errmsg, headers):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	361 # XXX The server can force infinite recursion here!
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	362 if headers.has_key('location'):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	363 newurl = headers['location']
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	364 elif headers.has_key('uri'):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	365 newurl = headers['uri']
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	366 else:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	367 return
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	368 void = fp.read()
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	369 fp.close()
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	370 return self.open(newurl)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	371
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	372 # Error 301 -- also relocated (permanently)
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	373 http_error_301 = http_error_302
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	374
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	375 # Error 401 -- authentication required
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	376 # See this URL for a description of the basic authentication scheme:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	377 # http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-v10-spec-00.txt
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	378 def http_error_401(self, url, fp, errcode, errmsg, headers):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	379 if headers.has_key('www-authenticate'):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	380 stuff = headers['www-authenticate']
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	381 p = regex.compile(
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	382 '[ \t]$[^ \t]+$[ \t]+realm="$[^"]$"')
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	383 if p.match(stuff) >= 0:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	384 scheme, realm = p.group(1, 2)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	385 if string.lower(scheme) == 'basic':
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	386 return self.retry_http_basic_auth(
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	387 url, realm)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	388
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	389 def retry_http_basic_auth(self, url, realm):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	390 host, selector = splithost(url)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	391 i = string.find(host, '@') + 1
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	392 host = host[i:]
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	393 user, passwd = self.get_user_passwd(host, realm, i)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	394 if not (user or passwd): return None
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	395 host = user + ':' + passwd + '@' + host
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	396 newurl = '//' + host + selector
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	397 return self.open_http(newurl)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	398
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	399 def get_user_passwd(self, host, realm, clear_cache = 0):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	400 key = realm + '@' + string.lower(host)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	401 if self.auth_cache.has_key(key):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	402 if clear_cache:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	403 del self.auth_cache[key]
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	404 else:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	405 return self.auth_cache[key]
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	406 user, passwd = self.prompt_user_passwd(host, realm)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	407 if user or passwd: self.auth_cache[key] = (user, passwd)
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	408 return user, passwd
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	409
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	410 def prompt_user_passwd(self, host, realm):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	411 # Override this in a GUI environment!
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	412 try:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	413 user = raw_input("Enter username for %s at %s: " %
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	414 (realm, host))
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	415 self.echo_off()
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	416 try:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	417 passwd = raw_input(
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	418 "Enter password for %s in %s at %s: " %
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	419 (user, realm, host))
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	420 finally:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	421 self.echo_on()
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	422 return user, passwd
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	423 except KeyboardInterrupt:
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	424 return None, None
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	425
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	426 def echo_off(self):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	427 import os
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	428 os.system("stty -echo")
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	429
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	430 def echo_on(self):
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	431 import os
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	432 print
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	433 os.system("stty echo")
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	434
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	435
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	436 # Utility functions
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	437
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	438 # Return the IP address of the magic hostname 'localhost'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	439 _localhost = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	440 def localhost():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	441 global _localhost
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	442 if not _localhost:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	443 _localhost = socket.gethostbyname('localhost')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	444 return _localhost
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	445
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	446 # Return the IP address of the current host
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	447 _thishost = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	448 def thishost():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	449 global _thishost
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	450 if not _thishost:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	451 _thishost = socket.gethostbyname(socket.gethostname())
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	452 return _thishost
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	453
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	454 # Return the set of errors raised by the FTP class
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	455 _ftperrors = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	456 def ftperrors():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	457 global _ftperrors
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	458 if not _ftperrors:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	459 import ftplib
5726 57a7ba1412e2 Catch all errors that ftplib can raise (ftplib.all_errors) rather Guido van Rossum <guido@python.org> parents: 5709 diff changeset	460 _ftperrors = ftplib.all_errors
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	461 return _ftperrors
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	462
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	463 # Return an empty mimetools.Message object
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	464 _noheaders = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	465 def noheaders():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	466 global _noheaders
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	467 if not _noheaders:
2899 65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	468 import mimetools
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	469 import StringIO
65f06fc39960 use mimetools; add error handling and authentication Guido van Rossum <guido@python.org> parents: 2779 diff changeset	470 _noheaders = mimetools.Message(StringIO.StringIO(), 0)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	471 _noheaders.fp.close() # Recycle file descriptor
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	472 return _noheaders
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	473
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	474
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	475 # Utility classes
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	476
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	477 # Class used by open_ftp() for cache of open FTP connections
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	478 class ftpwrapper:
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	479 def __init__(self, user, passwd, host, port, dirs):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	480 self.user = unquote(user or '')
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	481 self.passwd = unquote(passwd or '')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	482 self.host = host
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	483 self.port = port
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	484 self.dirs = []
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	485 for dir in dirs:
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	486 self.dirs.append(unquote(dir))
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	487 self.init()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	488 def init(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	489 import ftplib
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	490 self.ftp = ftplib.FTP()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	491 self.ftp.connect(self.host, self.port)
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	492 self.ftp.login(self.user, self.passwd)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	493 for dir in self.dirs:
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	494 self.ftp.cwd(dir)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	495 def retrfile(self, file, type):
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	496 import ftplib
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	497 if type in ('d', 'D'): cmd = 'TYPE A'; isdir = 1
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	498 else: cmd = 'TYPE ' + type; isdir = 0
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	499 try:
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	500 self.ftp.voidcmd(cmd)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	501 except ftplib.all_errors:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	502 self.init()
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	503 self.ftp.voidcmd(cmd)
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	504 conn = None
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	505 if file and not isdir:
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	506 try:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	507 cmd = 'RETR ' + file
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	508 conn = self.ftp.transfercmd(cmd)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	509 except ftplib.error_perm, reason:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	510 if reason[:3] != '550':
4558 df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	511 raise IOError, ('ftp error', reason), \
df9d1ebcf7f3 When re-raising an exception raised by a module used internally as Guido van Rossum <guido@python.org> parents: 4532 diff changeset	512 sys.exc_traceback
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	513 if not conn:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	514 # Try a directory listing
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	515 if file: cmd = 'LIST ' + file
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	516 else: cmd = 'LIST'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	517 conn = self.ftp.transfercmd(cmd)
5727 e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	518 return addclosehook(conn.makefile('rb'), self.endtransfer)
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	519 def endtransfer(self):
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	520 try:
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	521 self.ftp.voidresp()
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	522 except ftperrors():
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	523 pass
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	524 def close(self):
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	525 try:
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	526 self.ftp.close()
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	527 except ftperrors():
e0da3d89d4b6 Clear the ftp cache when it contains more than 10 entries. Guido van Rossum <guido@python.org> parents: 5726 diff changeset	528 pass
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	529
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	530 # Base class for addinfo and addclosehook
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	531 class addbase:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	532 def __init__(self, fp):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	533 self.fp = fp
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	534 self.read = self.fp.read
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	535 self.readline = self.fp.readline
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	536 self.readlines = self.fp.readlines
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	537 self.fileno = self.fp.fileno
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	538 def __repr__(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	539 return '<%s at %s whose fp = %s>' % (
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	540 self.__class__.__name__, `id(self)`, `self.fp`)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	541 def close(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	542 self.read = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	543 self.readline = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	544 self.readlines = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	545 self.fileno = None
2779 af7d0fea3fb2 add User-agent hdr; read and close the file upon http error Guido van Rossum <guido@python.org> parents: 2047 diff changeset	546 if self.fp: self.fp.close()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	547 self.fp = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	548
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	549 # Class to add a close hook to an open file
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	550 class addclosehook(addbase):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	551 def __init__(self, fp, closehook, *hookargs):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	552 addbase.__init__(self, fp)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	553 self.closehook = closehook
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	554 self.hookargs = hookargs
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	555 def close(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	556 if self.closehook:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	557 apply(self.closehook, self.hookargs)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	558 self.closehook = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	559 self.hookargs = None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	560 addbase.close(self)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	561
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	562 # class to add an info() method to an open file
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	563 class addinfo(addbase):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	564 def __init__(self, fp, headers):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	565 addbase.__init__(self, fp)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	566 self.headers = headers
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	567 def info(self):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	568 return self.headers
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	569
4257 239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	570 # class to add info() and geturl() methods to an open file
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	571 class addinfourl(addbase):
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	572 def __init__(self, fp, headers, url):
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	573 addbase.__init__(self, fp)
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	574 self.headers = headers
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	575 self.url = url
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	576 def info(self):
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	577 return self.headers
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	578 def geturl(self):
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	579 return self.url
239b10081176 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5. Guido van Rossum <guido@python.org> parents: 4153 diff changeset	580
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	581
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	582 # Utility to combine a URL with a base URL to form a new URL
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	583
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	584 def basejoin(base, url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	585 type, path = splittype(url)
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	586 if type:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	587 # if url is complete (i.e., it contains a type), return it
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	588 return url
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	589 host, path = splithost(path)
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	590 type, basepath = splittype(base) # inherit type from base
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	591 if host:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	592 # if url contains host, just inherit type
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	593 if type: return type + '://' + host + path
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	594 else:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	595 # no type inherited, so url must have started with //
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	596 # just return it
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	597 return url
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	598 host, basepath = splithost(basepath) # inherit host
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	599 basepath, basetag = splittag(basepath) # remove extraneuous cruft
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	600 basepath, basequery = splitquery(basepath) # idem
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	601 if path[:1] != '/':
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	602 # non-absolute path name
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	603 if path[:1] in ('#', '?'):
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	604 # path is just a tag or query, attach to basepath
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	605 i = len(basepath)
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	606 else:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	607 # else replace last component
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	608 i = string.rfind(basepath, '/')
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	609 if i < 0:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	610 # basepath not absolute
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	611 if host:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	612 # host present, make absolute
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	613 basepath = '/'
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	614 else:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	615 # else keep non-absolute
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	616 basepath = ''
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	617 else:
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	618 # remove last file component
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	619 basepath = basepath[:i+1]
5307 c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	620 # Interpret ../ (important because of symlinks)
c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	621 while basepath and path[:3] == '../':
c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	622 path = path[3:]
c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	623 i = string.rfind(basepath, '/')
c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	624 if i > 0:
c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	625 basepath = basepath[:i-1]
c3ed299d593e (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. Guido van Rossum <guido@python.org> parents: 5306 diff changeset	626
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	627 path = basepath + path
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	628 if type and host: return type + '://' + host + path
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	629 elif type: return type + ':' + path
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	630 elif host: return '//' + host + path # don't know what this means
2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	631 else: return path
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	632
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	633
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	634 # Utilities to parse URLs (most of these return None for missing parts):
3244 2842ffa52126 Fixed basejoin. There were two main problems: Sjoerd Mullender <sjoerd@acm.org> parents: 3028 diff changeset	635 # unwrap('<URL:type://host/path>') --> 'type://host/path'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	636 # splittype('type:opaquestring') --> 'type', 'opaquestring'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	637 # splithost('//host[:port]/path') --> 'host[:port]', '/path'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	638 # splituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	639 # splitpasswd('user:passwd') -> 'user', 'passwd'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	640 # splitport('host:port') --> 'host', 'port'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	641 # splitquery('/path?query') --> '/path', 'query'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	642 # splittag('/path#tag') --> '/path', 'tag'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	643 # splitattr('/path;attr1=value1;attr2=value2;...') ->
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	644 # '/path', ['attr1=value1', 'attr2=value2', ...]
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	645 # splitvalue('attr=value') --> 'attr', 'value'
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	646 # splitgophertype('/Xselector') --> 'X', 'selector'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	647 # unquote('abc%20def') -> 'abc def'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	648 # quote('abc def') -> 'abc%20def')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	649
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	650 def unwrap(url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	651 url = string.strip(url)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	652 if url[:1] == '<' and url[-1:] == '>':
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	653 url = string.strip(url[1:-1])
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	654 if url[:4] == 'URL:': url = string.strip(url[4:])
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	655 return url
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	656
5334 d1ac51b870e2 spliturl() should not throw away everything past first newline Guido van Rossum <guido@python.org> parents: 5307 diff changeset	657 _typeprog = regex.compile('^$[^/:]+$:')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	658 def splittype(url):
5334 d1ac51b870e2 spliturl() should not throw away everything past first newline Guido van Rossum <guido@python.org> parents: 5307 diff changeset	659 if _typeprog.match(url) >= 0:
d1ac51b870e2 spliturl() should not throw away everything past first newline Guido van Rossum <guido@python.org> parents: 5307 diff changeset	660 scheme = _typeprog.group(1)
d1ac51b870e2 spliturl() should not throw away everything past first newline Guido van Rossum <guido@python.org> parents: 5307 diff changeset	661 return scheme, url[len(scheme) + 1:]
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	662 return None, url
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	663
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	664 _hostprog = regex.compile('^//$[^/]+$$.*$$')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	665 def splithost(url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	666 if _hostprog.match(url) >= 0: return _hostprog.group(1, 2)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	667 return None, url
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	668
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	669 _userprog = regex.compile('^$[^@]$@$.$$')
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	670 def splituser(host):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	671 if _userprog.match(host) >= 0: return _userprog.group(1, 2)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	672 return None, host
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	673
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	674 _passwdprog = regex.compile('^$[^:]$:$.$$')
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	675 def splitpasswd(user):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	676 if _passwdprog.match(user) >= 0: return _passwdprog.group(1, 2)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	677 return user, None
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	678
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	679 _portprog = regex.compile('^$.*$:$[0-9]+$$')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	680 def splitport(host):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	681 if _portprog.match(host) >= 0: return _portprog.group(1, 2)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	682 return host, None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	683
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	684 # Split host and port, returning numeric port.
b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	685 # Return given default port if no ':' found; defaults to -1.
3687 d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	686 # Return numerical port if a valid number are found after ':'.
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	687 # Return None if ':' but not a valid number.
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	688 _nportprog = regex.compile('^$.$:$.$$')
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	689 def splitnport(host, defport=-1):
b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	690 if _nportprog.match(host) >= 0:
3687 d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	691 host, port = _nportprog.group(1, 2)
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	692 try:
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	693 if not port: raise string.atoi_error, "no digits"
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	694 nport = string.atoi(port)
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	695 except string.atoi_error:
d8f9dfc76bbf Change defn of splitnport() to only accept valid digit strings. Guido van Rossum <guido@python.org> parents: 3677 diff changeset	696 nport = None
3677 b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	697 return host, nport
b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	698 return host, defport
b05315816579 Added splitnport(), which is like splitport() but returns a numeric port, Guido van Rossum <guido@python.org> parents: 3660 diff changeset	699
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	700 _queryprog = regex.compile('^$.$\?$[^?]$$')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	701 def splitquery(url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	702 if _queryprog.match(url) >= 0: return _queryprog.group(1, 2)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	703 return url, None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	704
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	705 _tagprog = regex.compile('^$.$#$[^#]$$')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	706 def splittag(url):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	707 if _tagprog.match(url) >= 0: return _tagprog.group(1, 2)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	708 return url, None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	709
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	710 def splitattr(url):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	711 words = string.splitfields(url, ';')
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	712 return words[0], words[1:]
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	713
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	714 _valueprog = regex.compile('^$[^=]$=$.$$')
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	715 def splitvalue(attr):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	716 if _valueprog.match(attr) >= 0: return _valueprog.group(1, 2)
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	717 return attr, None
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	718
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	719 def splitgophertype(selector):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	720 if selector[:1] == '/' and selector[1:2]:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	721 return selector[1], selector[2:]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	722 return None, selector
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	723
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	724 _quoteprog = regex.compile('%[0-9a-fA-F][0-9a-fA-F]')
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	725 def unquote(s):
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	726 i = 0
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	727 n = len(s)
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	728 res = []
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	729 while 0 <= i < n:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	730 j = _quoteprog.search(s, i)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	731 if j < 0:
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	732 res.append(s[i:])
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	733 break
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	734 res.append(s[i:j] + chr(string.atoi(s[j+1:j+3], 16)))
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	735 i = j+3
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	736 return string.joinfields(res, '')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	737
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	738 def unquote_plus(s):
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	739 if '+' in s:
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	740 import regsub
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	741 s = regsub.gsub('+', ' ', s)
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	742 return unquote(s)
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	743
1828 b464e1d0b2fb New way of generating .pyc files, thanks to Sjoerd. Guido van Rossum <guido@python.org> parents: 1810 diff changeset	744 always_safe = string.letters + string.digits + '_,.-'
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	745 def quote(s, safe = '/'):
f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	746 safe = always_safe + safe
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	747 res = []
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	748 for c in s:
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	749 if c in safe:
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	750 res.append(c)
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	751 else:
4146 c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	752 res.append('%%%02x' % ord(c))
c4d66c410c83 Slightly faster (un)quoting. Guido van Rossum <guido@python.org> parents: 3724 diff changeset	753 return string.joinfields(res, '')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	754
4715 67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	755 def quote_plus(s, safe = '/'):
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	756 if ' ' in s:
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	757 import regsub
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	758 s = regsub.gsub(' ', '+', s)
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	759 return quote(s, safe + '+')
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	760 else:
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	761 return quote(s, safe)
67759dfe6ee8 Added quote_plus() and unquote_plus(), to do space/plus substitutions Guido van Rossum <guido@python.org> parents: 4659 diff changeset	762
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	763
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	764 # Proxy handling
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	765 def getproxies():
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	766 """Return a dictionary of protocol scheme -> proxy server URL mappings.
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	767
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	768 Scan the environment for variables named <scheme>_proxy;
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	769 this seems to be the standard convention. If you need a
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	770 different way, you can pass a proxies dictionary to the
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	771 [Fancy]URLopener constructor.
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	772
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	773 """
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	774 proxies = {}
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	775 for name, value in os.environ.items():
5653 309602c87929 lowercase proxies env variables, for Windows. Guido van Rossum <guido@python.org> parents: 5334 diff changeset	776 name = string.lower(name)
3461 803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	777 if value and name[-6:] == '_proxy':
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	778 proxies[name[:-6]] = value
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	779 return proxies
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	780
803fbcdcfc9f Added proxy handling; upped version. Guido van Rossum <guido@python.org> parents: 3397 diff changeset	781
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	782 # Test and time quote() and unquote()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	783 def test1():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	784 import time
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	785 s = ''
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	786 for i in range(256): s = s + chr(i)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	787 s = s*4
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	788 t0 = time.time()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	789 qs = quote(s)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	790 uqs = unquote(qs)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	791 t1 = time.time()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	792 if uqs != s:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	793 print 'Wrong!'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	794 print `s`
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	795 print `qs`
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	796 print `uqs`
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	797 print round(t1 - t0, 3), 'sec'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	798
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	799
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	800 # Test program
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	801 def test():
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	802 import sys
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	803 import regsub
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	804 args = sys.argv[1:]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	805 if not args:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	806 args = [
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	807 '/etc/passwd',
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	808 'file:/etc/passwd',
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	809 'file://localhost/etc/passwd',
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	810 'ftp://ftp.cwi.nl/etc/passwd',
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	811 'gopher://gopher.cwi.nl/11/',
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	812 'http://www.cwi.nl/index.html',
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	813 ]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	814 try:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	815 for url in args:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	816 print '-'10, url, '-'10
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	817 fn, h = urlretrieve(url)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	818 print fn, h
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	819 if h:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	820 print '======'
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	821 for k in h.keys(): print k + ':', h[k]
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	822 print '======'
5306 e614bcd34e34 Open files in binary mode. Guido van Rossum <guido@python.org> parents: 5241 diff changeset	823 fp = open(fn, 'rb')
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	824 data = fp.read()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	825 del fp
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	826 print regsub.gsub('\r', '', data)
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	827 fn, h = None, None
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	828 print '-'*40
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	829 finally:
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	830 urlcleanup()
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	831
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	832 # Run test program when run as a script
b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	833 if __name__ == '__main__':
1701 f1b040bca3e1 * Lib/urllib.py: implemented new quoting rules; added splituser, Guido van Rossum <guido@python.org> parents: 1590 diff changeset	834 ## test1()
1578 b21f048c23e9 Renamed urlopen.py to urllib.py. Guido van Rossum <guido@python.org> parents: diff changeset	835 test()

Mercurial > cpython-lru_cache-weakref

annotate Lib/urllib.py @ 5817:48b94de85bf3 legacy-trunk