Mercurial > cpython-lru_cache-weakref

annotate Lib/urlparse.py @ 13163:a09e8635534d legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Don't try to guess the name of a .def file -- if one is supplied, use it, otherwise just generate an '/export:' option.
author Greg Ward <gward@python.net>
date Thu, 29 Jun 2000 23:09:20 +0000
parents d115a91ca5fa
children db69c092c906
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11707
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
1 """Parse (absolute and relative) URLs.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
2
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
3 See RFC 1808: "Relative Uniform Resource Locators", by R. Fielding,
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
4 UC Irvine, June 1995.
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
5 """
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
7 # Standard/builtin Python modules
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8 import string
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
9 from string import join, split, rfind
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 # A classification of schemes ('' means apply by default)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 uses_relative = ['ftp', 'http', 'gopher', 'nntp', 'wais', 'file',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
13 'https', 'shttp',
12405
15554d3a68c7 Anthony Baxter <anthony@interlink.com.au>:
Fred Drake <fdrake@acm.org>
parents: 12336
diff changeset
14 'prospero', 'rtsp', 'rtspu', '']
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
15 uses_netloc = ['ftp', 'http', 'gopher', 'nntp', 'telnet', 'wais',
10264
b9bb75ded804 Steve Clift pointed out that 'file' allows a netloc.
Guido van Rossum <guido@python.org>
parents: 9793
diff changeset
16 'file',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
17 'https', 'shttp', 'snews',
12405
15554d3a68c7 Anthony Baxter <anthony@interlink.com.au>:
Fred Drake <fdrake@acm.org>
parents: 12336
diff changeset
18 'prospero', 'rtsp', 'rtspu', '']
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
19 non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', 'telnet', 'wais',
12967
d115a91ca5fa Pekka Pessi <Pekka.Pessi@nokia.com>:
Fred Drake <fdrake@acm.org>
parents: 12405
diff changeset
20 'snews', 'sip',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
21 ]
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
22 uses_params = ['ftp', 'hdl', 'prospero', 'http',
12967
d115a91ca5fa Pekka Pessi <Pekka.Pessi@nokia.com>:
Fred Drake <fdrake@acm.org>
parents: 12405
diff changeset
23 'https', 'shttp', 'rtsp', 'rtspu', 'sip',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
24 '']
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
25 uses_query = ['http', 'wais',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
26 'https', 'shttp',
12967
d115a91ca5fa Pekka Pessi <Pekka.Pessi@nokia.com>:
Fred Drake <fdrake@acm.org>
parents: 12405
diff changeset
27 'gopher', 'rtsp', 'rtspu', 'sip',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
28 '']
3641
e4faadb4abe2 added hdl protocol properties
Guido van Rossum <guido@python.org>
parents: 3482
diff changeset
29 uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', 'nntp', 'wais',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
30 'https', 'shttp', 'snews',
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31 'file', 'prospero', '']
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
32
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
33 # Characters valid in scheme names
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
34 scheme_chars = string.letters + string.digits + '+-.'
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
35
5785
b247297be9a0 After some discussion with Jeremy and Fred, decided to limit the
Guido van Rossum <guido@python.org>
parents: 5776
diff changeset
36 MAX_CACHE_SIZE = 20
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
37 _parse_cache = {}
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
38
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
39 def clear_cache():
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
40 """Clear the parse cache."""
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
41 global _parse_cache
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
42 _parse_cache = {}
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
43
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
44
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
45 def urlparse(url, scheme = '', allow_fragments = 1):
11707
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
46 """Parse a URL into 6 components:
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
47 <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
48 Return a 6-tuple: (scheme, netloc, path, params, query, fragment).
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
49 Note that we don't break the components up in smaller bits
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
50 (e.g. netloc is a single string) and we don't expand % escapes."""
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
51 key = url, scheme, allow_fragments
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
52 cached = _parse_cache.get(key, None)
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
53 if cached:
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
54 return cached
4831
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
55 if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
56 clear_cache()
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
57 find = string.find
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
58 netloc = path = params = query = fragment = ''
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
59 i = find(url, ':')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
60 if i > 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
61 if url[:i] == 'http': # optimize the common case
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
62 scheme = string.lower(url[:i])
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
63 url = url[i+1:]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
64 if url[:2] == '//':
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
65 i = find(url, '/', 2)
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
66 if i < 0:
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
67 i = len(url)
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
68 netloc = url[2:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
69 url = url[i:]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
70 if allow_fragments:
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
71 i = string.rfind(url, '#')
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
72 if i >= 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
73 fragment = url[i+1:]
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
74 url = url[:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
75 i = find(url, '?')
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
76 if i >= 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
77 query = url[i+1:]
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
78 url = url[:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
79 i = find(url, ';')
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
80 if i >= 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
81 params = url[i+1:]
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
82 url = url[:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
83 tuple = scheme, netloc, url, params, query, fragment
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
84 _parse_cache[key] = tuple
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
85 return tuple
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 for c in url[:i]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
87 if c not in scheme_chars:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
89 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
90 scheme, url = string.lower(url[:i]), url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
91 if scheme in uses_netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
92 if url[:2] == '//':
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
93 i = find(url, '/', 2)
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
94 if i < 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
95 i = len(url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
96 netloc, url = url[2:i], url[i:]
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
97 if allow_fragments and scheme in uses_fragment:
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
98 i = string.rfind(url, '#')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
99 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
100 url, fragment = url[:i], url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 if scheme in uses_query:
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
102 i = find(url, '?')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
103 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
104 url, query = url[:i], url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
105 if scheme in uses_params:
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
106 i = find(url, ';')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
107 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
108 url, params = url[:i], url[i+1:]
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
109 tuple = scheme, netloc, url, params, query, fragment
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
110 _parse_cache[key] = tuple
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
111 return tuple
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
112
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
113 def urlunparse((scheme, netloc, url, params, query, fragment)):
11707
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
114 """Put a parsed URL back together again. This may result in a
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
115 slightly different, but equivalent URL, if the URL that was parsed
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
116 originally had redundant delimiters, e.g. a ? with an empty query
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
117 (the draft states that these are equivalent)."""
10421
0d6bd391acd8 Sjoerd Mullender writes:
Guido van Rossum <guido@python.org>
parents: 10416
diff changeset
118 if netloc or (scheme in uses_netloc and url[:2] == '//'):
2900
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
119 if url[:1] != '/': url = '/' + url
10421
0d6bd391acd8 Sjoerd Mullender writes:
Guido van Rossum <guido@python.org>
parents: 10416
diff changeset
120 url = '//' + (netloc or '') + url
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 if scheme:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122 url = scheme + ':' + url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 if params:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 url = url + ';' + params
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 if query:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 url = url + '?' + query
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
127 if fragment:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
128 url = url + '#' + fragment
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 return url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
130
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
131 def urljoin(base, url, allow_fragments = 1):
11707
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
132 """Join a base URL and a possibly relative URL to form an absolute
6f4146846337 The third and final doc-string sweep by Ka-Ping Yee.
Guido van Rossum <guido@python.org>
parents: 10830
diff changeset
133 interpretation of the latter."""
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
134 if not base:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 return url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
136 bscheme, bnetloc, bpath, bparams, bquery, bfragment = \
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
137 urlparse(base, '', allow_fragments)
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
138 scheme, netloc, path, params, query, fragment = \
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
139 urlparse(url, bscheme, allow_fragments)
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
140 if scheme != bscheme or scheme not in uses_relative:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
141 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
142 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
143 if scheme in uses_netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
144 if netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
145 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
146 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 netloc = bnetloc
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 if path[:1] == '/':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
149 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
150 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
151 if not path:
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
152 return urlunparse((scheme, netloc, bpath,
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
153 params, query or bquery, fragment))
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
154 segments = split(bpath, '/')[:-1] + split(path, '/')
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
155 # XXX The stuff below is bogus in various ways...
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
156 if segments[-1] == '.':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
157 segments[-1] = ''
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
158 while '.' in segments:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
159 segments.remove('.')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
160 while 1:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
161 i = 1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
162 n = len(segments) - 1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
163 while i < n:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
164 if segments[i] == '..' and segments[i-1]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
165 del segments[i-1:i+1]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
166 break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
167 i = i+1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
169 break
6737
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
170 if len(segments) == 2 and segments[1] == '..' and segments[0] == '':
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
171 segments[-1] = ''
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
172 elif len(segments) >= 2 and segments[-1] == '..':
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 segments[-2:] = ['']
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
174 return urlunparse((scheme, netloc, join(segments, '/'),
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
176
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
177 def urldefrag(url):
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
178 """Removes any existing fragment from URL.
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
179
12336
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
180 Returns a tuple of the defragmented URL and the fragment. If
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
181 the URL contained no fragments, the second element is the
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
182 empty string.
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
183 """
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
184 s, n, p, a, q, frag = urlparse(url)
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
185 defrag = urlunparse((s, n, p, a, q, ''))
d33743102f8a Some cleanup -- don't use splitfields/joinfields, standardize
Guido van Rossum <guido@python.org>
parents: 11707
diff changeset
186 return defrag, frag
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
187
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
188
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
189 test_input = """
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
190 http://a/b/c/d
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
191
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
192 g:h = <URL:g:h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
193 http:g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
194 http: = <URL:http://a/b/c/d>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
195 g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
196 ./g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
197 g/ = <URL:http://a/b/c/g/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
198 /g = <URL:http://a/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
199 //g = <URL:http://g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
200 ?y = <URL:http://a/b/c/d?y>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
201 g?y = <URL:http://a/b/c/g?y>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
202 g?y/./x = <URL:http://a/b/c/g?y/./x>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
203 . = <URL:http://a/b/c/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
204 ./ = <URL:http://a/b/c/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 .. = <URL:http://a/b/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 ../ = <URL:http://a/b/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
207 ../g = <URL:http://a/b/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 ../.. = <URL:http://a/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 ../../g = <URL:http://a/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
210 ../../../g = <URL:http://a/../g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 ./../g = <URL:http://a/b/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 ./g/. = <URL:http://a/b/c/g/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
213 /./g = <URL:http://a/./g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 g/./h = <URL:http://a/b/c/g/h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 g/../h = <URL:http://a/b/c/h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 http:g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
217 http: = <URL:http://a/b/c/d>
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
218 http:?y = <URL:http://a/b/c/d?y>
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
219 http:g?y = <URL:http://a/b/c/g?y>
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
220 http:g?y/./x = <URL:http://a/b/c/g?y/./x>
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
221 """
9665
e11d75005a5a Add XXX comment about a test that doesn't seem right -- no time to
Guido van Rossum <guido@python.org>
parents: 9068
diff changeset
222 # XXX The result for //g is actually http://g/; is this a problem?
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
223
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
224 def test():
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
225 import sys
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 base = ''
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 if sys.argv[1:]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
228 fn = sys.argv[1]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
229 if fn == '-':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
230 fp = sys.stdin
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
231 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
232 fp = open(fn)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
233 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
234 import StringIO
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
235 fp = StringIO.StringIO(test_input)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
236 while 1:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
237 line = fp.readline()
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
238 if not line: break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
239 words = string.split(line)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
240 if not words:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
241 continue
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
242 url = words[0]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
243 parts = urlparse(url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
244 print '%-10s : %s' % (url, parts)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
245 abs = urljoin(base, url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
246 if not base:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
247 base = abs
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
248 wrapped = '<URL:%s>' % abs
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
249 print '%-10s = %s' % (url, wrapped)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
250 if len(words) == 3 and words[1] == '=':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
251 if wrapped != words[2]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 print 'EXPECTED', words[2], '!!!!!!!!!!'
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
253
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
254 if __name__ == '__main__':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
255 test()