Mercurial > cpython-lru_cache-weakref

annotate Lib/urlparse.py @ 8831:00a6bb65ff29 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Getting rid of this ancient stuff.
author Guido van Rossum <guido@python.org>
date Wed, 05 Aug 1998 21:10:09 +0000
parents de4c81c06994
children 839c6622d15a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3482
00ca900c7742 Update reference (it's now RFC 1808); added http to list of protocols
Guido van Rossum <guido@python.org>
parents: 2962
diff changeset
1 # Parse (absolute and relative) URLs. See RFC 1808: "Relative Uniform
00ca900c7742 Update reference (it's now RFC 1808); added http to list of protocols
Guido van Rossum <guido@python.org>
parents: 2962
diff changeset
2 # Resource Locators", by R. Fielding, UC Irvine, June 1995.
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
3
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
4 # Standard/builtin Python modules
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 import string
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
6 from string import joinfields, splitfields, find, rfind
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
7
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8 # A classification of schemes ('' means apply by default)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
9 uses_relative = ['ftp', 'http', 'gopher', 'nntp', 'wais', 'file',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
10 'https', 'shttp',
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 'prospero', '']
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 uses_netloc = ['ftp', 'http', 'gopher', 'nntp', 'telnet', 'wais',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
13 'https', 'shttp', 'snews',
2962
f72abd62d508 remove file: from list of protocols taking host
Guido van Rossum <guido@python.org>
parents: 2900
diff changeset
14 'prospero', '']
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
15 non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', 'telnet', 'wais',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
16 'snews',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
17 ]
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
18 uses_params = ['ftp', 'hdl', 'prospero', 'http',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
19 'https', 'shttp',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
20 '']
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
21 uses_query = ['http', 'wais',
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
22 'https', 'shttp',
7125
de4c81c06994 Add Gopher to list of protocols that support query strings.
Guido van Rossum <guido@python.org>
parents: 6737
diff changeset
23 'gopher',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
24 '']
3641
e4faadb4abe2 added hdl protocol properties
Guido van Rossum <guido@python.org>
parents: 3482
diff changeset
25 uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', 'nntp', 'wais',
4853
cf90745381bf Added characteristics of shttp, https, and snews.
Guido van Rossum <guido@python.org>
parents: 4831
diff changeset
26 'https', 'shttp', 'snews',
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 'file', 'prospero', '']
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
28
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
29 # Characters valid in scheme names
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
30 scheme_chars = string.letters + string.digits + '+-.'
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31
5785
b247297be9a0 After some discussion with Jeremy and Fred, decided to limit the
Guido van Rossum <guido@python.org>
parents: 5776
diff changeset
32 MAX_CACHE_SIZE = 20
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
33 _parse_cache = {}
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
34
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
35 def clear_cache():
4831
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
36 """Clear the parse cache."""
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
37 global _parse_cache
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
38 _parse_cache = {}
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
39
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
40
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 # Parse a URL into 6 components:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
42 # <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 # Return a 6-tuple: (scheme, netloc, path, params, query, fragment).
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
44 # Note that we don't break the components up in smaller bits
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
45 # (e.g. netloc is a single string) and we don't expand % escapes.
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
46 def urlparse(url, scheme = '', allow_framents = 1):
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
47 key = url, scheme, allow_framents
5776
f6de05005576 Test urlparse cache with try/except instead of has_key.
Guido van Rossum <guido@python.org>
parents: 4853
diff changeset
48 try:
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
49 return _parse_cache[key]
5776
f6de05005576 Test urlparse cache with try/except instead of has_key.
Guido van Rossum <guido@python.org>
parents: 4853
diff changeset
50 except KeyError:
f6de05005576 Test urlparse cache with try/except instead of has_key.
Guido van Rossum <guido@python.org>
parents: 4853
diff changeset
51 pass
4831
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
52 if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth
d87e87853958 Crude but effective hack to clear the parser cache every so often.
Guido van Rossum <guido@python.org>
parents: 3652
diff changeset
53 clear_cache()
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
54 netloc = path = params = query = fragment = ''
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 i = string.find(url, ':')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
56 if i > 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
57 for c in url[:i]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 if c not in scheme_chars:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
60 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
61 scheme, url = string.lower(url[:i]), url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
62 if scheme in uses_netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
63 if url[:2] == '//':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
64 i = string.find(url, '/', 2)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
65 if i < 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
66 i = len(url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
67 netloc, url = url[2:i], url[i:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
68 if allow_framents and scheme in uses_fragment:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
69 i = string.rfind(url, '#')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
70 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
71 url, fragment = url[:i], url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
72 if scheme in uses_query:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
73 i = string.find(url, '?')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
74 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
75 url, query = url[:i], url[i+1:]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
76 if scheme in uses_params:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
77 i = string.find(url, ';')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
78 if i >= 0:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
79 url, params = url[:i], url[i+1:]
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
80 tuple = scheme, netloc, url, params, query, fragment
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
81 _parse_cache[key] = tuple
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
82 return tuple
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
83
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
84 # Put a parsed URL back together again. This may result in a slightly
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 # different, but equivalent URL, if the URL that was parsed originally
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 # had redundant delimiters, e.g. a ? with an empty query (the draft
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
87 # states that these are equivalent).
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 def urlunparse((scheme, netloc, url, params, query, fragment)):
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
89 if netloc:
2900
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
90 if url[:1] != '/': url = '/' + url
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
91 url = '//' + netloc + url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
92 if scheme:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
93 url = scheme + ':' + url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
94 if params:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
95 url = url + ';' + params
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
96 if query:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
97 url = url + '?' + query
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
98 if fragment:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
99 url = url + '#' + fragment
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
100 return url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
101
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
102 # Join a base URL and a possibly relative URL to form an absolute
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
103 # interpretation of the latter.
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
104 def urljoin(base, url, allow_framents = 1):
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
105 if not base:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
106 return url
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
107 bscheme, bnetloc, bpath, bparams, bquery, bfragment = \
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
108 urlparse(base, '', allow_framents)
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
109 scheme, netloc, path, params, query, fragment = \
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
110 urlparse(url, bscheme, allow_framents)
2047
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
111 # XXX Unofficial hack: default netloc to bnetloc even if
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
112 # schemes differ
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
113 if scheme != bscheme and not netloc and \
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
114 scheme in uses_relative and bscheme in uses_relative and \
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
115 scheme in uses_netloc and bscheme in uses_netloc:
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
116 netloc = bnetloc
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
117 # Strip the port number
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
118 i = find(netloc, '@')
2047
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
119 if i < 0: i = 0
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
120 i = find(netloc, ':', i)
2047
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
121 if i >= 0:
f18c1915c722 Add hacks for switching protocol and path but leaving host unchanged
Guido van Rossum <guido@python.org>
parents: 1875
diff changeset
122 netloc = netloc[:i]
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 if scheme != bscheme or scheme not in uses_relative:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 if scheme in uses_netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
127 if netloc:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
128 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
130 netloc = bnetloc
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 if path[:1] == '/':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 return urlunparse((scheme, netloc, path,
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
134 if not path:
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
135 return urlunparse((scheme, netloc, bpath,
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
136 params, query or bquery, fragment))
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
137 i = rfind(bpath, '/')
2900
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
138 if i >= 0:
0acbd85bcdba subtle changes to relative rurl joins
Guido van Rossum <guido@python.org>
parents: 2047
diff changeset
139 path = bpath[:i] + '/' + path
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
140 segments = splitfields(path, '/')
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
141 if segments[-1] == '.':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
142 segments[-1] = ''
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
143 while '.' in segments:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
144 segments.remove('.')
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
145 while 1:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
146 i = 1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 n = len(segments) - 1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 while i < n:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
149 if segments[i] == '..' and segments[i-1]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
150 del segments[i-1:i+1]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
151 break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
152 i = i+1
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
153 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
154 break
6737
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
155 if len(segments) == 2 and segments[1] == '..' and segments[0] == '':
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
156 segments[-1] = ''
ab3d7a28e07d Patch my Marc Lemburg to fix urljoin("/a", "..") and urljoin("/a", "..#1").
Guido van Rossum <guido@python.org>
parents: 5785
diff changeset
157 elif len(segments) >= 2 and segments[-1] == '..':
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
158 segments[-2:] = ['']
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
159 return urlunparse((scheme, netloc, joinfields(segments, '/'),
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
160 params, query, fragment))
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
161
3652
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
162 def urldefrag(url):
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
163 """Removes any existing fragment from URL.
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
164
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
165 Returns a tuple of the defragmented URL and the fragment. If
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
166 the URL contained no fragments, the second element is the
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
167 empty string.
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
168 """
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
169 s, n, p, a, q, frag = urlparse(url)
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
170 defrag = urlunparse((s, n, p, a, q, ''))
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
171 return defrag, frag
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
172
728a39a3e0d5 optimizations due to Fred Drake; added urldefrag() function
Guido van Rossum <guido@python.org>
parents: 3641
diff changeset
173
1875
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
174 test_input = """
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 http://a/b/c/d
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
176
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
177 g:h = <URL:g:h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
178 http:g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
179 http: = <URL:http://a/b/c/d>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
180 g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
181 ./g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
182 g/ = <URL:http://a/b/c/g/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 /g = <URL:http://a/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 //g = <URL:http://g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 ?y = <URL:http://a/b/c/d?y>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
186 g?y = <URL:http://a/b/c/g?y>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
187 g?y/./x = <URL:http://a/b/c/g?y/./x>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
188 . = <URL:http://a/b/c/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
189 ./ = <URL:http://a/b/c/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
190 .. = <URL:http://a/b/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
191 ../ = <URL:http://a/b/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
192 ../g = <URL:http://a/b/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
193 ../.. = <URL:http://a/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
194 ../../g = <URL:http://a/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
195 ../../../g = <URL:http://a/../g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
196 ./../g = <URL:http://a/b/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
197 ./g/. = <URL:http://a/b/c/g/>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
198 /./g = <URL:http://a/./g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
199 g/./h = <URL:http://a/b/c/g/h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
200 g/../h = <URL:http://a/b/c/h>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
201 http:g = <URL:http://a/b/c/g>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
202 http: = <URL:http://a/b/c/d>
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
203 """
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
204
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 def test():
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 import sys
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
207 base = ''
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 if sys.argv[1:]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 fn = sys.argv[1]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
210 if fn == '-':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 fp = sys.stdin
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
213 fp = open(fn)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 else:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 import StringIO
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 fp = StringIO.StringIO(test_input)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
217 while 1:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
218 line = fp.readline()
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
219 if not line: break
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
220 words = string.split(line)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
221 if not words:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
222 continue
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
223 url = words[0]
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
224 parts = urlparse(url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
225 print '%-10s : %s' % (url, parts)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 abs = urljoin(base, url)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 if not base:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
228 base = abs
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
229 wrapped = '<URL:%s>' % abs
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
230 print '%-10s = %s' % (url, wrapped)
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
231 if len(words) == 3 and words[1] == '=':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
232 if wrapped != words[2]:
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
233 print 'EXPECTED', words[2], '!!!!!!!!!!'
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
234
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
235 if __name__ == '__main__':
8c9cc054c5ed New tty/pty modules by Steen; new urlparser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
236 test()