Mercurial > cpython-lru_cache-weakref

annotate Lib/json/decoder.py @ 87690:66c1c9f32567 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859
author Radomir Dopieralski <devel@sheep.art.pl>
date Tue, 03 Dec 2013 16:02:48 +0100
parents 5f7326ed850f
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
1 """Implementation of JSONDecoder
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
2 """
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
3 import re
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
4
70085
c2853a54b29e #5723: Improve json tests to be executed with and without accelerations.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 69319
diff changeset
5 from json import scanner
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
6 try:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
7 from _json import scanstring as c_scanstring
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
8 except ImportError:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
9 c_scanstring = None
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
10
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
11 __all__ = ['JSONDecoder']
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
12
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
13 FLAGS = re.VERBOSE | re.MULTILINE | re.DOTALL
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
14
80633
402bae9f8fe5 Cleanup json decoder: float() has builtin support of nan, +inf, -inf since Python 2.6
Victor Stinner <victor.stinner@gmail.com>
parents: 77841
diff changeset
15 NaN = float('nan')
402bae9f8fe5 Cleanup json decoder: float() has builtin support of nan, +inf, -inf since Python 2.6
Victor Stinner <victor.stinner@gmail.com>
parents: 77841
diff changeset
16 PosInf = float('inf')
402bae9f8fe5 Cleanup json decoder: float() has builtin support of nan, +inf, -inf since Python 2.6
Victor Stinner <victor.stinner@gmail.com>
parents: 77841
diff changeset
17 NegInf = float('-inf')
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
18
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
19
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
20 def linecol(doc, pos):
49524
00e467db6182 merge r66932 and add a few py3k only checks
Benjamin Peterson <benjamin@python.org>
parents: 48769
diff changeset
21 if isinstance(doc, bytes):
00e467db6182 merge r66932 and add a few py3k only checks
Benjamin Peterson <benjamin@python.org>
parents: 48769
diff changeset
22 newline = b'\n'
00e467db6182 merge r66932 and add a few py3k only checks
Benjamin Peterson <benjamin@python.org>
parents: 48769
diff changeset
23 else:
00e467db6182 merge r66932 and add a few py3k only checks
Benjamin Peterson <benjamin@python.org>
parents: 48769
diff changeset
24 newline = '\n'
00e467db6182 merge r66932 and add a few py3k only checks
Benjamin Peterson <benjamin@python.org>
parents: 48769
diff changeset
25 lineno = doc.count(newline, 0, pos) + 1
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
26 if lineno == 1:
82301
36220cf535aa Issue #17225: JSON decoder now counts columns in the first line starting
Serhiy Storchaka <storchaka@gmail.com>
parents: 77840
diff changeset
27 colno = pos + 1
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
28 else:
49524
00e467db6182 merge r66932 and add a few py3k only checks
Benjamin Peterson <benjamin@python.org>
parents: 48769
diff changeset
29 colno = pos - doc.rindex(newline, 0, pos)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
30 return lineno, colno
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
31
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
32
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
33 def errmsg(msg, doc, pos, end=None):
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
34 # Note that this function is called from _json
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
35 lineno, colno = linecol(doc, pos)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
36 if end is None:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
37 fmt = '{0}: line {1} column {2} (char {3})'
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
38 return fmt.format(msg, lineno, colno, pos)
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
39 #fmt = '%s: line %d column %d (char %d)'
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
40 #return fmt % (msg, lineno, colno, pos)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
41 endlineno, endcolno = linecol(doc, end)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
42 fmt = '{0}: line {1} column {2} - line {3} column {4} (char {5} - {6})'
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
43 return fmt.format(msg, lineno, colno, endlineno, endcolno, pos, end)
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
44 #fmt = '%s: line %d column %d - line %d column %d (char %d - %d)'
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
45 #return fmt % (msg, lineno, colno, endlineno, endcolno, pos, end)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
46
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
47
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
48 _CONSTANTS = {
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
49 '-Infinity': NegInf,
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
50 'Infinity': PosInf,
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
51 'NaN': NaN,
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
52 }
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
53
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
54
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
55 STRINGCHUNK = re.compile(r'(.*?)(["\\\x00-\x1f])', FLAGS)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
56 BACKSLASH = {
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
57 '"': '"', '\\': '\\', '/': '/',
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
58 'b': '\b', 'f': '\f', 'n': '\n', 'r': '\r', 't': '\t',
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
59 }
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
60
87588
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
61 def _decode_uXXXX(s, pos):
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
62 esc = s[pos + 1:pos + 5]
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
63 if len(esc) == 4 and esc[1] not in 'xX':
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
64 try:
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
65 return int(esc, 16)
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
66 except ValueError:
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
67 pass
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
68 msg = "Invalid \\uXXXX escape"
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
69 raise ValueError(errmsg(msg, s, pos))
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
70
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
71 def py_scanstring(s, end, strict=True,
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
72 _b=BACKSLASH, _m=STRINGCHUNK.match):
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
73 """Scan the string s for a JSON string. End is the index of the
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
74 character in s after the quote that started the JSON string.
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
75 Unescapes all valid JSON string escape sequences and raises ValueError
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
76 on attempt to decode an invalid string. If strict is False then literal
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
77 control characters are allowed in the string.
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
78
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
79 Returns a tuple of the decoded string and the index of the character in s
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
80 after the end quote."""
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
81 chunks = []
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
82 _append = chunks.append
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
83 begin = end - 1
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
84 while 1:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
85 chunk = _m(s, end)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
86 if chunk is None:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
87 raise ValueError(
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
88 errmsg("Unterminated string starting at", s, begin))
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
89 end = chunk.end()
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
90 content, terminator = chunk.groups()
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
91 # Content is contains zero or more unescaped string characters
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
92 if content:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
93 _append(content)
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
94 # Terminator is the end of string, a literal control character,
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
95 # or a backslash denoting that an escape sequence follows
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
96 if terminator == '"':
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
97 break
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
98 elif terminator != '\\':
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
99 if strict:
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
100 #msg = "Invalid control character %r at" % (terminator,)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
101 msg = "Invalid control character {0!r} at".format(terminator)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
102 raise ValueError(errmsg(msg, s, end))
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
103 else:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
104 _append(terminator)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
105 continue
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
106 try:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
107 esc = s[end]
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
108 except IndexError:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
109 raise ValueError(
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
110 errmsg("Unterminated string starting at", s, begin))
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
111 # If not a unicode escape sequence, must be in the lookup table
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
112 if esc != 'u':
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
113 try:
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
114 char = _b[esc]
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
115 except KeyError:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
116 msg = "Invalid \\escape: {0!r}".format(esc)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
117 raise ValueError(errmsg(msg, s, end))
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
118 end += 1
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
119 else:
87588
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
120 uni = _decode_uXXXX(s, end)
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
121 end += 5
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
122 if 0xd800 <= uni <= 0xdbff and s[end:end + 2] == '\\u':
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
123 uni2 = _decode_uXXXX(s, end + 1)
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
124 if 0xdc00 <= uni2 <= 0xdfff:
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
125 uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00))
8abbdbe86c01 Issue #11489: JSON decoder now accepts lone surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 82635
diff changeset
126 end += 6
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
127 char = chr(uni)
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
128 _append(char)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
129 return ''.join(chunks), end
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
130
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
131
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
132 # Use speedup if available
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
133 scanstring = c_scanstring or py_scanstring
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
134
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
135 WHITESPACE = re.compile(r'[ \t\n\r]*', FLAGS)
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
136 WHITESPACE_STR = ' \t\n\r'
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
137
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
138
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
139 def JSONObject(s_and_end, strict, scan_once, object_hook, object_pairs_hook,
64443
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
140 memo=None, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
141 s, end = s_and_end
53296
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
142 pairs = []
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
143 pairs_append = pairs.append
64443
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
144 # Backwards compatibility
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
145 if memo is None:
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
146 memo = {}
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
147 memo_get = memo.setdefault
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
148 # Use a slice to prevent IndexError from being raised, the following
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
149 # check will raise a more specific ValueError if the string is empty
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
150 nextchar = s[end:end + 1]
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
151 # Normally we expect nextchar == '"'
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
152 if nextchar != '"':
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
153 if nextchar in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
154 end = _w(s, end).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
155 nextchar = s[end:end + 1]
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
156 # Trivial empty object
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
157 if nextchar == '}':
69319
a220458179ed #9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
Ezio Melotti
parents: 69317
diff changeset
158 if object_pairs_hook is not None:
a220458179ed #9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
Ezio Melotti
parents: 69317
diff changeset
159 result = object_pairs_hook(pairs)
82634
40c36d873f41 #17368: Fix an off-by-one error in the Python JSON decoder that caused a failure while decoding empty object literals when object_pairs_hook was specified.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 82301
diff changeset
160 return result, end + 1
69319
a220458179ed #9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
Ezio Melotti
parents: 69317
diff changeset
161 pairs = {}
a220458179ed #9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
Ezio Melotti
parents: 69317
diff changeset
162 if object_hook is not None:
a220458179ed #9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
Ezio Melotti
parents: 69317
diff changeset
163 pairs = object_hook(pairs)
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
164 return pairs, end + 1
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
165 elif nextchar != '"':
77840
9854520c8200 Issue #5067: improve some json error messages.
Antoine Pitrou <solipsis@pitrou.net>
parents: 70090
diff changeset
166 raise ValueError(errmsg(
9854520c8200 Issue #5067: improve some json error messages.
Antoine Pitrou <solipsis@pitrou.net>
parents: 70090
diff changeset
167 "Expecting property name enclosed in double quotes", s, end))
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
168 end += 1
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
169 while True:
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
170 key, end = scanstring(s, end, strict)
64443
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
171 key = memo_get(key, key)
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
172 # To skip some function call overhead we optimize the fast paths where
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
173 # the JSON key separator is ": " or just ":".
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
174 if s[end:end + 1] != ':':
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
175 end = _w(s, end).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
176 if s[end:end + 1] != ':':
77840
9854520c8200 Issue #5067: improve some json error messages.
Antoine Pitrou <solipsis@pitrou.net>
parents: 70090
diff changeset
177 raise ValueError(errmsg("Expecting ':' delimiter", s, end))
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
178 end += 1
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
179
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
180 try:
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
181 if s[end] in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
182 end += 1
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
183 if s[end] in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
184 end = _w(s, end + 1).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
185 except IndexError:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
186 pass
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
187
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
188 try:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
189 value, end = scan_once(s, end)
81270
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
190 except StopIteration as err:
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
191 raise ValueError(errmsg("Expecting value", s, err.value)) from None
53296
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
192 pairs_append((key, value))
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
193 try:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
194 nextchar = s[end]
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
195 if nextchar in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
196 end = _w(s, end + 1).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
197 nextchar = s[end]
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
198 except IndexError:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
199 nextchar = ''
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
200 end += 1
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
201
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
202 if nextchar == '}':
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
203 break
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
204 elif nextchar != ',':
77840
9854520c8200 Issue #5067: improve some json error messages.
Antoine Pitrou <solipsis@pitrou.net>
parents: 70090
diff changeset
205 raise ValueError(errmsg("Expecting ',' delimiter", s, end - 1))
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
206 end = _w(s, end).end()
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
207 nextchar = s[end:end + 1]
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
208 end += 1
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
209 if nextchar != '"':
77840
9854520c8200 Issue #5067: improve some json error messages.
Antoine Pitrou <solipsis@pitrou.net>
parents: 70090
diff changeset
210 raise ValueError(errmsg(
9854520c8200 Issue #5067: improve some json error messages.
Antoine Pitrou <solipsis@pitrou.net>
parents: 70090
diff changeset
211 "Expecting property name enclosed in double quotes", s, end - 1))
53296
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
212 if object_pairs_hook is not None:
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
213 result = object_pairs_hook(pairs)
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
214 return result, end
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
215 pairs = dict(pairs)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
216 if object_hook is not None:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
217 pairs = object_hook(pairs)
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
218 return pairs, end
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
219
69317
7019fc1a9663 #9233: Fix json to work properly even when _json is not available.
Ezio Melotti
parents: 66470
diff changeset
220 def JSONArray(s_and_end, scan_once, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
221 s, end = s_and_end
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
222 values = []
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
223 nextchar = s[end:end + 1]
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
224 if nextchar in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
225 end = _w(s, end + 1).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
226 nextchar = s[end:end + 1]
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
227 # Look-ahead for trivial empty array
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
228 if nextchar == ']':
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
229 return values, end + 1
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
230 _append = values.append
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
231 while True:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
232 try:
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
233 value, end = scan_once(s, end)
81270
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
234 except StopIteration as err:
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
235 raise ValueError(errmsg("Expecting value", s, err.value)) from None
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
236 _append(value)
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
237 nextchar = s[end:end + 1]
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
238 if nextchar in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
239 end = _w(s, end + 1).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
240 nextchar = s[end:end + 1]
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
241 end += 1
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
242 if nextchar == ']':
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
243 break
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
244 elif nextchar != ',':
81270
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
245 raise ValueError(errmsg("Expecting ',' delimiter", s, end - 1))
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
246 try:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
247 if s[end] in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
248 end += 1
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
249 if s[end] in _ws:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
250 end = _w(s, end + 1).end()
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
251 except IndexError:
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
252 pass
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
253
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
254 return values, end
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
255
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
256
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
257 class JSONDecoder(object):
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
258 """Simple JSON <http://json.org> decoder
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
259
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
260 Performs the following translations in decoding by default:
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
261
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
262 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
263 | JSON | Python |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
264 +===============+===================+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
265 | object | dict |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
266 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
267 | array | list |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
268 +---------------+-------------------+
63575
4874efd1f657 #9087: update json docstrings -- unicode and long do not exist anymore.
Georg Brandl <georg@python.org>
parents: 53667
diff changeset
269 | string | str |
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
270 +---------------+-------------------+
63575
4874efd1f657 #9087: update json docstrings -- unicode and long do not exist anymore.
Georg Brandl <georg@python.org>
parents: 53667
diff changeset
271 | number (int) | int |
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
272 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
273 | number (real) | float |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
274 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
275 | true | True |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
276 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
277 | false | False |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
278 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
279 | null | None |
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
280 +---------------+-------------------+
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
281
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
282 It also understands ``NaN``, ``Infinity``, and ``-Infinity`` as
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
283 their corresponding ``float`` values, which is outside the JSON spec.
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
284
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
285 """
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
286
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
287 def __init__(self, object_hook=None, parse_float=None,
53296
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
288 parse_int=None, parse_constant=None, strict=True,
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
289 object_pairs_hook=None):
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
290 """``object_hook``, if specified, will be called with the result
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
291 of every JSON object decoded and its return value will be used in
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
292 place of the given ``dict``. This can be used to provide custom
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
293 deserializations (e.g. to support JSON-RPC class hinting).
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
294
66470
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
295 ``object_pairs_hook``, if specified will be called with the result of
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
296 every JSON object decoded with an ordered list of pairs. The return
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
297 value of ``object_pairs_hook`` will be used instead of the ``dict``.
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
298 This feature can be used to implement custom decoders that rely on the
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
299 order that the key and value pairs are decoded (for example,
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
300 collections.OrderedDict will remember the order of insertion). If
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
301 ``object_hook`` is also defined, the ``object_pairs_hook`` takes
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
302 priority.
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
303
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
304 ``parse_float``, if specified, will be called with the string
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
305 of every JSON float to be decoded. By default this is equivalent to
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
306 float(num_str). This can be used to use another datatype or parser
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
307 for JSON floats (e.g. decimal.Decimal).
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
308
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
309 ``parse_int``, if specified, will be called with the string
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
310 of every JSON int to be decoded. By default this is equivalent to
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
311 int(num_str). This can be used to use another datatype or parser
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
312 for JSON integers (e.g. float).
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
313
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
314 ``parse_constant``, if specified, will be called with one of the
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
315 following strings: -Infinity, Infinity, NaN.
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
316 This can be used to raise an exception if invalid JSON numbers
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
317 are encountered.
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
318
66470
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
319 If ``strict`` is false (true is the default), then control
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
320 characters will be allowed inside strings. Control characters in
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
321 this context are those with character codes in the 0-31 range,
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
322 including ``'\\t'`` (tab), ``'\\n'``, ``'\\r'`` and ``'\\0'``.
45e49269e0d8 Merged revisions 85530,85532-85534,85538-85543,85546-85548 via svnmerge from
Georg Brandl <georg@python.org>
parents: 63575
diff changeset
323
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
324 """
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
325 self.object_hook = object_hook
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
326 self.parse_float = parse_float or float
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
327 self.parse_int = parse_int or int
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
328 self.parse_constant = parse_constant or _CONSTANTS.__getitem__
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
329 self.strict = strict
53296
93c929a2fac7 Forward port r70471: Add object_pairs_hook. Issue 5381.
Raymond Hettinger <python@rcn.com>
parents: 49524
diff changeset
330 self.object_pairs_hook = object_pairs_hook
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
331 self.parse_object = JSONObject
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
332 self.parse_array = JSONArray
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
333 self.parse_string = scanstring
64443
c3ad883b940b Issue #7451: Improve decoding performance of JSON objects, and reduce
Antoine Pitrou <solipsis@pitrou.net>
parents: 63575
diff changeset
334 self.memo = {}
70085
c2853a54b29e #5723: Improve json tests to be executed with and without accelerations.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 69319
diff changeset
335 self.scan_once = scanner.make_scanner(self)
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
336
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
337
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
338 def decode(self, s, _w=WHITESPACE.match):
63575
4874efd1f657 #9087: update json docstrings -- unicode and long do not exist anymore.
Georg Brandl <georg@python.org>
parents: 53667
diff changeset
339 """Return the Python representation of ``s`` (a ``str`` instance
4874efd1f657 #9087: update json docstrings -- unicode and long do not exist anymore.
Georg Brandl <georg@python.org>
parents: 53667
diff changeset
340 containing a JSON document).
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
341
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
342 """
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
343 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
344 end = _w(s, end).end()
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
345 if end != len(s):
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
346 raise ValueError(errmsg("Extra data", s, end, len(s)))
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
347 return obj
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
348
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
349 def raw_decode(self, s, idx=0):
63575
4874efd1f657 #9087: update json docstrings -- unicode and long do not exist anymore.
Georg Brandl <georg@python.org>
parents: 53667
diff changeset
350 """Decode a JSON document from ``s`` (a ``str`` beginning with
4874efd1f657 #9087: update json docstrings -- unicode and long do not exist anymore.
Georg Brandl <georg@python.org>
parents: 53667
diff changeset
351 a JSON document) and return a 2-tuple of the Python
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
352 representation and the index in ``s`` where the document ended.
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
353
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
354 This can be used to decode a JSON document from a string that may
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
355 have extraneous data at the end.
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
356
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
357 """
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
358 try:
53667
f686aced02a3 port simplejson upgrade from the trunk #4136
Benjamin Peterson <benjamin@python.org>
parents: 53296
diff changeset
359 obj, end = self.scan_once(s, idx)
81270
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
360 except StopIteration as err:
17b4bf9e9f43 #16009: JSON error messages now provide more information. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 80633
diff changeset
361 raise ValueError(errmsg("Expecting value", s, err.value)) from None
46737
9f45979558a9 Merged revisions 62734,62736,62748,62769 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents:
diff changeset
362 return obj, end