Mercurial > cpython-lru_cache-weakref

annotate Lib/urlparse.py @ 10123:bc287c6cfa23 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Restructured a bit.
author Guido van Rossum <guido@python.org>
date Mon, 08 Feb 1999 22:27:18 +0000
parents 737d05a9bb5e
children b9bb75ded804
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3482
00ca900c7742 Update reference (it's now RFC 1808); added http to list of protocols
Guido van Rossum <guido@python.org>
parents: 2962
diff changeset
1 # Parse (absolute and relative) URLs. See RFC 1808: "Relative Uniform
00ca900c7742 Update reference (it's now RFC 1808); added http to list of protocols
Guido van Rossum <guido@python.org>
parents: 2962
diff changeset
2 # Resource Locators", by R. Fielding, UC Irvine, June 1995.
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
3
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
4 # Standard/builtin Python modules
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 import string
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
6 from string import joinfields, splitfields, find, rfind
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
7
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8 # A classification of schemes ('' means apply by default)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
9 uses_relative = ['ftp', 'http', 'gopher', 'nntp', 'wais', 'file',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
10 'https', 'shttp',
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 'prospero', '']
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 uses_netloc = ['ftp', 'http', 'gopher', 'nntp', 'telnet', 'wais',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
13 'https', 'shttp', 'snews',
2962
f72abd62d508 remove file: from list of protocols taking host
Guido van Rossum <guido@python.org>
parents: 2900
diff changeset
14 'prospero', '']
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
15 non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', 'telnet', 'wais',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
16 'snews',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
17 ]
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
18 uses_params = ['ftp', 'hdl', 'prospero', 'http',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
19 'https', 'shttp',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
20 '']
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
21 uses_query = ['http', 'wais',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
22 'https', 'shttp',
7125
de4c81c06994 Add Gopher to list of protocols that support query strings.
Guido van Rossum <guido@python.org>
parents: 6737
diff changeset
23 'gopher',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
24 '']
3641
e4faadb4abe2 added hdl protocol properties
Guido van Rossum <guido@python.org>
parents: 3482
diff changeset
25 uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', 'nntp', 'wais',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
26 'https', 'shttp', 'snews',
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 'file', 'prospero', '']
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
28
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
29 # Characters valid in scheme names
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
30 scheme_chars = string.letters + string.digits + '+-.'
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31
5785
b247297be9a0 After some discussion with Jeremy and Fred, decided to limit the
Guido van Rossum <guido@python.org>
parents: 5776
diff changeset
32 MAX_CACHE_SIZE = 20
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
33 _parse_cache = {}
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
34
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
35 def clear_cache():
4831
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
36 """Clear the parse cache."""
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
37 global _parse_cache
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
38 _parse_cache = {}
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
39
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
40
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 # Parse a URL into 6 components:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
42 # <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 # Return a 6-tuple: (scheme, netloc, path, params, query, fragment).
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
44 # Note that we don't break the components up in smaller bits
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
45 # (e.g. netloc is a single string) and we don't expand % escapes.
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
46 def urlparse(url, scheme = '', allow_fragments = 1):
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
47 key = url, scheme, allow_fragments
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
48 cached = _parse_cache.get(key, None)
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
49 if cached:
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
50 return cached
4831
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
51 if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
52 clear_cache()
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
53 find = string.find
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
54 netloc = path = params = query = fragment = ''
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
55 i = find(url, ':')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
56 if i > 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
57 if url[:i] == 'http': # optimize the common case
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
58 scheme = string.lower(url[:i])
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
59 url = url[i+1:]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
60 if url[:2] == '//':
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
61 i = find(url, '/', 2)
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
62 if i < 0:
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
63 i = len(url)
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
64 netloc = url[2:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
65 url = url[i:]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
66 if allow_fragments:
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
67 i = string.rfind(url, '#')
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
68 if i >= 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
69 fragment = url[i+1:]
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
70 url = url[:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
71 i = find(url, '?')
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
72 if i >= 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
73 query = url[i+1:]
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
74 url = url[:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
75 i = find(url, ';')
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
76 if i >= 0:
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
77 params = url[i+1:]
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
78 url = url[:i]
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
79 tuple = scheme, netloc, url, params, query, fragment
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
80 _parse_cache[key] = tuple
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
81 return tuple
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
82 for c in url[:i]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
83 if c not in scheme_chars:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
84 break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 scheme, url = string.lower(url[:i]), url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
87 if scheme in uses_netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 if url[:2] == '//':
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
89 i = find(url, '/', 2)
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
90 if i < 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
91 i = len(url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
92 netloc, url = url[2:i], url[i:]
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
93 if allow_fragments and scheme in uses_fragment:
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
94 i = string.rfind(url, '#')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
95 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
96 url, fragment = url[:i], url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
97 if scheme in uses_query:
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
98 i = find(url, '?')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
99 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
100 url, query = url[:i], url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 if scheme in uses_params:
9068
b25d5a62e357 Easy optimizations of urlparse for the common case of parsing an http URL.
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 9051
diff changeset
102 i = find(url, ';')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
103 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
104 url, params = url[:i], url[i+1:]
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
105 tuple = scheme, netloc, url, params, query, fragment
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
106 _parse_cache[key] = tuple
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
107 return tuple
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
108
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
109 # Put a parsed URL back together again. This may result in a slightly
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
110 # different, but equivalent URL, if the URL that was parsed originally
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
111 # had redundant delimiters, e.g. a ? with an empty query (the draft
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
112 # states that these are equivalent).
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
113 def urlunparse((scheme, netloc, url, params, query, fragment)):
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
114 if netloc:
2900
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
115 if url[:1] != '/': url = '/' + url
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
116 url = '//' + netloc + url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
117 if scheme:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
118 url = scheme + ':' + url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 if params:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
120 url = url + ';' + params
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 if query:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122 url = url + '?' + query
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 if fragment:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 url = url + '#' + fragment
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 return url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
127 # Join a base URL and a possibly relative URL to form an absolute
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
128 # interpretation of the latter.
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
129 def urljoin(base, url, allow_fragments = 1):
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
130 if not base:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 return url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 bscheme, bnetloc, bpath, bparams, bquery, bfragment = \
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
133 urlparse(base, '', allow_fragments)
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
134 scheme, netloc, path, params, query, fragment = \
9051
839c6622d15a fix typo in keyword argument 'allow_frament' should be 'allow_fragment'
Jeremy Hylton <jeremy@alum.mit.edu>
parents: 7125
diff changeset
135 urlparse(url, bscheme, allow_fragments)
2047
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
136 # XXX Unofficial hack: default netloc to bnetloc even if
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
137 # schemes differ
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
138 if scheme != bscheme and not netloc and \
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
139 scheme in uses_relative and bscheme in uses_relative and \
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
140 scheme in uses_netloc and bscheme in uses_netloc:
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
141 netloc = bnetloc
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
142 # Strip the port number
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
143 i = find(netloc, '@')
2047
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
144 if i < 0: i = 0
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
145 i = find(netloc, ':', i)
2047
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
146 if i >= 0:
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
147 netloc = netloc[:i]
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 if scheme != bscheme or scheme not in uses_relative:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
149 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
150 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
151 if scheme in uses_netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
152 if netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
153 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
154 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
155 netloc = bnetloc
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
156 if path[:1] == '/':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
157 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
158 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
159 if not path:
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
160 return urlunparse((scheme, netloc, bpath,
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
161 params, query or bquery, fragment))
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
162 i = rfind(bpath, '/')
2900
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
163 if i >= 0:
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
164 path = bpath[:i] + '/' + path
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
165 segments = splitfields(path, '/')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
166 if segments[-1] == '.':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
167 segments[-1] = ''
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 while '.' in segments:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
169 segments.remove('.')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
170 while 1:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
171 i = 1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
172 n = len(segments) - 1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 while i < n:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
174 if segments[i] == '..' and segments[i-1]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 del segments[i-1:i+1]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
176 break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
177 i = i+1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
178 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
179 break
6737
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
180 if len(segments) == 2 and segments[1] == '..' and segments[0] == '':
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
181 segments[-1] = ''
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
182 elif len(segments) >= 2 and segments[-1] == '..':
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 segments[-2:] = ['']
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
184 return urlunparse((scheme, netloc, joinfields(segments, '/'),
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
186
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
187 def urldefrag(url):
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
188 """Removes any existing fragment from URL.
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
189
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
190 Returns a tuple of the defragmented URL and the fragment. If
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
191 the URL contained no fragments, the second element is the
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
192 empty string.
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
193 """
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
194 s, n, p, a, q, frag = urlparse(url)
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
195 defrag = urlunparse((s, n, p, a, q, ''))
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
196 return defrag, frag
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
197
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
198
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
199 test_input = """
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
200 http://a/b/c/d
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
201
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
202 g:h = <URL:g:h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
203 http:g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
204 http: = <URL:http://a/b/c/d>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 ./g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
207 g/ = <URL:http://a/b/c/g/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 /g = <URL:http://a/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 //g = <URL:http://g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
210 ?y = <URL:http://a/b/c/d?y>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 g?y = <URL:http://a/b/c/g?y>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 g?y/./x = <URL:http://a/b/c/g?y/./x>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
213 . = <URL:http://a/b/c/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 ./ = <URL:http://a/b/c/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 .. = <URL:http://a/b/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 ../ = <URL:http://a/b/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
217 ../g = <URL:http://a/b/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
218 ../.. = <URL:http://a/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
219 ../../g = <URL:http://a/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
220 ../../../g = <URL:http://a/../g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
221 ./../g = <URL:http://a/b/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
222 ./g/. = <URL:http://a/b/c/g/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
223 /./g = <URL:http://a/./g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
224 g/./h = <URL:http://a/b/c/g/h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
225 g/../h = <URL:http://a/b/c/h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 http:g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 http: = <URL:http://a/b/c/d>
9793
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
228 http:?y = <URL:http://a/b/c/d?y>
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
229 http:g?y = <URL:http://a/b/c/g?y>
737d05a9bb5e Fixed bug in the common-case code for HTTP URLs; it would lose the query,
Andrew M. Kuchling <amk@amk.ca>
parents: 9665
diff changeset
230 http:g?y/./x = <URL:http://a/b/c/g?y/./x>
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
231 """
9665
e11d75005a5a Add XXX comment about a test that doesn't seem right -- no time to
Guido van Rossum <guido@python.org>
parents: 9068
diff changeset
232 # XXX The result for //g is actually http://g/; is this a problem?
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
233
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
234 def test():
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
235 import sys
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
236 base = ''
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
237 if sys.argv[1:]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
238 fn = sys.argv[1]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
239 if fn == '-':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
240 fp = sys.stdin
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
241 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
242 fp = open(fn)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
243 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
244 import StringIO
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
245 fp = StringIO.StringIO(test_input)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
246 while 1:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
247 line = fp.readline()
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
248 if not line: break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
249 words = string.split(line)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
250 if not words:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
251 continue
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 url = words[0]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
253 parts = urlparse(url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
254 print '%-10s : %s' % (url, parts)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
255 abs = urljoin(base, url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
256 if not base:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
257 base = abs
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
258 wrapped = '<URL:%s>' % abs
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
259 print '%-10s = %s' % (url, wrapped)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
260 if len(words) == 3 and words[1] == '=':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
261 if wrapped != words[2]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
262 print 'EXPECTED', words[2], '!!!!!!!!!!'
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
263
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
264 if __name__ == '__main__':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
265 test()