Mercurial > cpython-lru_cache-weakref

annotate Include/pyatomic.h @ 81922:4ab386b00aaf

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix tests for issue #11159.
author Serhiy Storchaka <storchaka@gmail.com>
date Sat, 02 Feb 2013 12:17:50 +0200
parents 4579cd952156
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
66664
62b61abd02b8 Merge branches/pep-0384.
Martin v. Löwis <martin@v.loewis.de>
parents: 61184
diff changeset
1 #ifndef Py_LIMITED_API
61184
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
2 #ifndef Py_ATOMIC_H
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
3 #define Py_ATOMIC_H
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
4 /* XXX: When compilers start offering a stdatomic.h with lock-free
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
5 atomic_int and atomic_address types, include that here and rewrite
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
6 the atomic operations in terms of it. */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
7
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
8 #include "dynamic_annotations.h"
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
9
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
10 #ifdef __cplusplus
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
11 extern "C" {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
12 #endif
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
13
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
14 /* This is modeled after the atomics interface from C1x, according to
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
15 * the draft at
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
16 * http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1425.pdf.
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
17 * Operations and types are named the same except with a _Py_ prefix
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
18 * and have the same semantics.
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
19 *
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
20 * Beware, the implementations here are deep magic.
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
21 */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
22
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
23 typedef enum _Py_memory_order {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
24 _Py_memory_order_relaxed,
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
25 _Py_memory_order_acquire,
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
26 _Py_memory_order_release,
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
27 _Py_memory_order_acq_rel,
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
28 _Py_memory_order_seq_cst
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
29 } _Py_memory_order;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
30
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
31 typedef struct _Py_atomic_address {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
32 void *_value;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
33 } _Py_atomic_address;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
34
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
35 typedef struct _Py_atomic_int {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
36 int _value;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
37 } _Py_atomic_int;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
38
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
39 /* Only support GCC (for expression statements) and x86 (for simple
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
40 * atomic semantics) for now */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
41 #if defined(__GNUC__) && (defined(__i386__) || defined(__amd64))
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
42
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
43 static __inline__ void
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
44 _Py_atomic_signal_fence(_Py_memory_order order)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
45 {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
46 if (order != _Py_memory_order_relaxed)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
47 __asm__ volatile("":::"memory");
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
48 }
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
49
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
50 static __inline__ void
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
51 _Py_atomic_thread_fence(_Py_memory_order order)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
52 {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
53 if (order != _Py_memory_order_relaxed)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
54 __asm__ volatile("mfence":::"memory");
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
55 }
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
56
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
57 /* Tell the race checker about this operation's effects. */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
58 static __inline__ void
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
59 _Py_ANNOTATE_MEMORY_ORDER(const volatile void *address, _Py_memory_order order)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
60 {
73866
4579cd952156 - Issue #11147: Fix an unused argument in _Py_ANNOTATE_MEMORY_ORDER. (Fix
Barry Warsaw <barry@python.org>
parents: 73624
diff changeset
61 (void)address; /* shut up -Wunused-parameter */
61184
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
62 switch(order) {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
63 case _Py_memory_order_release:
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
64 case _Py_memory_order_acq_rel:
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
65 case _Py_memory_order_seq_cst:
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
66 _Py_ANNOTATE_HAPPENS_BEFORE(address);
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
67 break;
73624
f855f929bc48 Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
Petri Lehtinen <petri@digip.org>
parents: 66664
diff changeset
68 case _Py_memory_order_relaxed:
f855f929bc48 Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
Petri Lehtinen <petri@digip.org>
parents: 66664
diff changeset
69 case _Py_memory_order_acquire:
61184
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
70 break;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
71 }
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
72 switch(order) {
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
73 case _Py_memory_order_acquire:
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
74 case _Py_memory_order_acq_rel:
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
75 case _Py_memory_order_seq_cst:
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
76 _Py_ANNOTATE_HAPPENS_AFTER(address);
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
77 break;
73624
f855f929bc48 Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
Petri Lehtinen <petri@digip.org>
parents: 66664
diff changeset
78 case _Py_memory_order_relaxed:
f855f929bc48 Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
Petri Lehtinen <petri@digip.org>
parents: 66664
diff changeset
79 case _Py_memory_order_release:
61184
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
80 break;
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
81 }
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
82 }
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
83
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
84 #define _Py_atomic_store_explicit(ATOMIC_VAL, NEW_VAL, ORDER) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
85 __extension__ ({ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
86 __typeof__(ATOMIC_VAL) atomic_val = ATOMIC_VAL; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
87 __typeof__(atomic_val->_value) new_val = NEW_VAL;\
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
88 volatile __typeof__(new_val) *volatile_data = &atomic_val->_value; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
89 _Py_memory_order order = ORDER; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
90 _Py_ANNOTATE_MEMORY_ORDER(atomic_val, order); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
91 \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
92 /* Perform the operation. */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
93 _Py_ANNOTATE_IGNORE_WRITES_BEGIN(); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
94 switch(order) { \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
95 case _Py_memory_order_release: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
96 _Py_atomic_signal_fence(_Py_memory_order_release); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
97 /* fallthrough */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
98 case _Py_memory_order_relaxed: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
99 *volatile_data = new_val; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
100 break; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
101 \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
102 case _Py_memory_order_acquire: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
103 case _Py_memory_order_acq_rel: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
104 case _Py_memory_order_seq_cst: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
105 __asm__ volatile("xchg %0, %1" \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
106 : "+r"(new_val) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
107 : "m"(atomic_val->_value) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
108 : "memory"); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
109 break; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
110 } \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
111 _Py_ANNOTATE_IGNORE_WRITES_END(); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
112 })
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
113
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
114 #define _Py_atomic_load_explicit(ATOMIC_VAL, ORDER) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
115 __extension__ ({ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
116 __typeof__(ATOMIC_VAL) atomic_val = ATOMIC_VAL; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
117 __typeof__(atomic_val->_value) result; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
118 volatile __typeof__(result) *volatile_data = &atomic_val->_value; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
119 _Py_memory_order order = ORDER; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
120 _Py_ANNOTATE_MEMORY_ORDER(atomic_val, order); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
121 \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
122 /* Perform the operation. */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
123 _Py_ANNOTATE_IGNORE_READS_BEGIN(); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
124 switch(order) { \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
125 case _Py_memory_order_release: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
126 case _Py_memory_order_acq_rel: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
127 case _Py_memory_order_seq_cst: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
128 /* Loads on x86 are not releases by default, so need a */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
129 /* thread fence. */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
130 _Py_atomic_thread_fence(_Py_memory_order_release); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
131 break; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
132 default: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
133 /* No fence */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
134 break; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
135 } \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
136 result = *volatile_data; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
137 switch(order) { \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
138 case _Py_memory_order_acquire: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
139 case _Py_memory_order_acq_rel: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
140 case _Py_memory_order_seq_cst: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
141 /* Loads on x86 are automatically acquire operations so */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
142 /* can get by with just a compiler fence. */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
143 _Py_atomic_signal_fence(_Py_memory_order_acquire); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
144 break; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
145 default: \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
146 /* No fence */ \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
147 break; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
148 } \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
149 _Py_ANNOTATE_IGNORE_READS_END(); \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
150 result; \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
151 })
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
152
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
153 #else /* !gcc x86 */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
154 /* Fall back to other compilers and processors by assuming that simple
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
155 volatile accesses are atomic. This is false, so people should port
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
156 this. */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
157 #define _Py_atomic_signal_fence(/*memory_order*/ ORDER) ((void)0)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
158 #define _Py_atomic_thread_fence(/*memory_order*/ ORDER) ((void)0)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
159 #define _Py_atomic_store_explicit(ATOMIC_VAL, NEW_VAL, ORDER) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
160 ((ATOMIC_VAL)->_value = NEW_VAL)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
161 #define _Py_atomic_load_explicit(ATOMIC_VAL, ORDER) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
162 ((ATOMIC_VAL)->_value)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
163
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
164 #endif /* !gcc x86 */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
165
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
166 /* Standardized shortcuts. */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
167 #define _Py_atomic_store(ATOMIC_VAL, NEW_VAL) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
168 _Py_atomic_store_explicit(ATOMIC_VAL, NEW_VAL, _Py_memory_order_seq_cst)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
169 #define _Py_atomic_load(ATOMIC_VAL) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
170 _Py_atomic_load_explicit(ATOMIC_VAL, _Py_memory_order_seq_cst)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
171
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
172 /* Python-local extensions */
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
173
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
174 #define _Py_atomic_store_relaxed(ATOMIC_VAL, NEW_VAL) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
175 _Py_atomic_store_explicit(ATOMIC_VAL, NEW_VAL, _Py_memory_order_relaxed)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
176 #define _Py_atomic_load_relaxed(ATOMIC_VAL) \
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
177 _Py_atomic_load_explicit(ATOMIC_VAL, _Py_memory_order_relaxed)
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
178
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
179 #ifdef __cplusplus
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
180 }
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
181 #endif
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
182
4fd3ab5fa71d Make (most of) Python's tests pass under Thread Sanitizer.
Jeffrey Yasskin <jyasskin@gmail.com>
parents:
diff changeset
183 #endif /* Py_ATOMIC_H */
66664
62b61abd02b8 Merge branches/pep-0384.
Martin v. Löwis <martin@v.loewis.de>
parents: 61184
diff changeset
184 #endif /* Py_LIMITED_API */