cpython-lru_cache-weakref: Include/objimpl.h history

Issue #18783: Removed existing mentions of Python long type in docstrings,

Tue, 27 Aug 2013 20:17:03 +0300

Issue #18783: Removed existing mentions of Python long type in docstrings,
error messages and comments.

Issue #18112: PEP 442 implementation (safe object finalization).

Tue, 30 Jul 2013 19:59:21 +0200

Issue #18112: PEP 442 implementation (safe object finalization).

Issue #3329: Implement the PEP 445

Sun, 07 Jul 2013 02:05:46 +0200

Issue #3329: Implement the PEP 445

Add new enum:

* PyMemAllocatorDomain

Add new structures:

* PyMemAllocator
* PyObjectArenaAllocator

Add new functions:

* PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree()
* PyMem_GetAllocator(), PyMem_SetAllocator()
* PyObject_GetArenaAllocator(), PyObject_SetArenaAllocator()
* PyMem_SetupDebugHooks()

Changes:

* PyMem_Malloc()/PyObject_Realloc() now always call malloc()/realloc(), instead
of calling PyObject_Malloc()/PyObject_Realloc() in debug mode.
* PyObject_Malloc()/PyObject_Realloc() now falls back to
PyMem_Malloc()/PyMem_Realloc() for allocations larger than 512 bytes.
* Redesign debug checks on memory block allocators as hooks, instead of using C
macros

Revert changeset 6661a8154eb3: Issue #3329: Add new APIs to customize memory allocators

Sat, 15 Jun 2013 03:37:01 +0200

Revert changeset 6661a8154eb3: Issue #3329: Add new APIs to customize memory allocators

The new API require more discussion.

Issue #3329: Add new APIs to customize memory allocators

Sat, 15 Jun 2013 00:37:46 +0200

Issue #3329: Add new APIs to customize memory allocators

* Add a new PyMemAllocators structure
* New functions:

- PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
allocator functions
- PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
- PyMem_GetAllocators(), PyMem_SetAllocators()
- PyMem_SetupDebugHooks()
- _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()

* Add unit test for PyMem_Malloc(0) and PyObject_Malloc(0)
* Add unit test for new get/set allocators functions
* PyObject_Malloc() now falls back on PyMem_Malloc() instead of malloc() if
size is bigger than SMALL_REQUEST_THRESHOLD, and PyObject_Realloc() falls
back on PyMem_Realloc() instead of realloc()
* PyMem_Malloc() and PyMem_Realloc() now always call malloc() and realloc(),
instead of calling PyObject_Malloc() and PyObject_Realloc() in debug mode

Issue #17937: Try harder to collect cyclic garbage at shutdown.

Sun, 19 May 2013 01:11:58 +0200

Issue #17937: Try harder to collect cyclic garbage at shutdown.

Using 'long double' to force this structure to be worst case aligned is no

Mon, 10 Dec 2012 18:05:05 -0800

Using 'long double' to force this structure to be worst case aligned is no
longer required as of Python 2.5+ when the gc_refs changed from an int (4
bytes) to a Py_ssize_t (8 bytes) as the minimum size is 16 bytes.

The use of a 'long double' triggered a warning by Clang trunk's
Undefined-Behavior Sanitizer as on many platforms a long double requires
16-byte alignment but the Python memory allocator only guarantees 8 byte
alignment.

So our code would allocate and use these structures with technically improper
alignment. Though it didn't matter since the 'dummy' field is never used.
This silences that warning.

Spelunking into code history, the double was added in 2001 to force better
alignment on some platforms and changed to a long double in 2002 to appease
Tru64. That issue should no loner be present since the upgrade from int to
Py_ssize_t where the minimum structure size increased to 16 (unless anyone
knows of a platform where ssize_t is 4 bytes?) or 24 bytes depending on if the
build uses 4 or 8 byte pointers.

We can probably get rid of the double and this union hack all together today.
That is a slightly more invasive change that can be left for later.

A more correct non-hacky alternative if any alignment issues are still found
would be to use a compiler specific alignment declaration on the structure and
determine which value to use at configure time.

Issue #13390: New function :func:`sys.getallocatedblocks()` returns the number of memory blocks currently allocated.

Sun, 09 Dec 2012 14:28:26 +0100

Issue #13390: New function :func:`sys.getallocatedblocks()` returns the number of memory blocks currently allocated.
Also, the ``-R`` option to regrtest uses this function to guard against memory allocation leaks.

Issue #18783: Removed existing mentions of Python long type in docstrings,

Tue, 27 Aug 2013 19:40:23 +0300

Issue #18783: Removed existing mentions of Python long type in docstrings,
error messages and comments.

Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t.

Thu, 20 Sep 2012 20:56:47 +0200

Issue #15144: Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t.
Patch by Serhiy Storchaka.

Issue #14785: Add sys._debugmallocstats() to help debug low-level memory allocation issues

Fri, 22 Jun 2012 14:55:41 -0400

Issue #14785: Add sys._debugmallocstats() to help debug low-level memory allocation issues

Dummy-merge 3.1 branch into 3.2 branch.

Sat, 05 Mar 2011 15:15:55 +0100

Dummy-merge 3.1 branch into 3.2 branch.

Issue #10333: Remove ancient GC API, which has been deprecated since

Tue, 04 Jan 2011 00:00:31 +0000

Issue #10333: Remove ancient GC API, which has been deprecated since
Python 2.2.

Merge branches/pep-0384.

Fri, 03 Dec 2010 20:14:31 +0000

Merge branches/pep-0384.

Recorded merge of revisions 81029 via svnmerge from

Sun, 09 May 2010 15:52:27 +0000

Recorded merge of revisions 81029 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines

Untabify C files. Will watch buildbots.
........

http://bugs.python.org/issue6836

Mon, 28 Sep 2009 13:45:02 +0000

http://bugs.python.org/issue6836
Merging revisions 75103,75104 from trunk to py3k

Recorded merge of revisions 81032 via svnmerge from

Sun, 09 May 2010 16:14:21 +0000

Recorded merge of revisions 81032 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
r81032 | antoine.pitrou | 2010-05-09 17:52:27 +0200 (dim., 09 mai 2010) | 9 lines

Recorded merge of revisions 81029 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines

Untabify C files. Will watch buildbots.
........
................

Merged revisions 70546 via svnmerge from

Mon, 23 Mar 2009 18:52:06 +0000

Merged revisions 70546 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
r70546 | antoine.pitrou | 2009-03-23 19:41:45 +0100 (lun., 23 mars 2009) | 9 lines

Issue #4688: Add a heuristic so that tuples and dicts containing only
untrackable objects are not tracked by the garbage collector. This can
reduce the size of collections and therefore the garbage collection overhead
on long-running programs, depending on their particular use of datatypes.

(trivia: this makes the "binary_trees" benchmark from the Computer Language
Shootout 40% faster)
........

#1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT.

Wed, 19 Dec 2007 02:45:37 +0000

#1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT.

Remove _PyObject_Del

Sun, 19 Aug 2007 22:48:23 +0000

Remove _PyObject_Del

Merged revisions 56467-56482 via svnmerge from

Sat, 21 Jul 2007 17:22:18 +0000

Merged revisions 56467-56482 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/p3yk

................
r56477 | martin.v.loewis | 2007-07-21 09:04:38 +0200 (Sa, 21 Jul 2007) | 11 lines

Merged revisions 56466-56476 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
r56476 | martin.v.loewis | 2007-07-21 08:55:02 +0200 (Sa, 21 Jul 2007) | 4 lines

PEP 3123: Provide forward compatibility with Python 3.0, while keeping
backwards compatibility. Add Py_Refcnt, Py_Type, Py_Size, and
PyVarObject_HEAD_INIT.
........
................
r56478 | martin.v.loewis | 2007-07-21 09:47:23 +0200 (Sa, 21 Jul 2007) | 2 lines

PEP 3123: Use proper C inheritance for PyObject.
................
r56479 | martin.v.loewis | 2007-07-21 10:06:55 +0200 (Sa, 21 Jul 2007) | 3 lines

Add longintrepr.h to Python.h, so that the compiler can
see that PyFalse is really some kind of PyObject*.
................
r56480 | martin.v.loewis | 2007-07-21 10:47:18 +0200 (Sa, 21 Jul 2007) | 2 lines

Qualify SHIFT, MASK, BASE.
................
r56482 | martin.v.loewis | 2007-07-21 19:10:57 +0200 (Sa, 21 Jul 2007) | 2 lines

Correctly refer to _ob_next.
................

Get rid of most of the flags (in tp_flags) that keep track of various

Thu, 27 Jul 2006 21:53:35 +0000

Get rid of most of the flags (in tp_flags) that keep track of various
variations of the type struct and its attachments. In Py3k, all type
structs have to have all fields -- no binary backwards compatibility.
Had to change the complex object to a new-style number!

Merge p3yk branch with the trunk up to revision 45595. This breaks a fair

Fri, 21 Apr 2006 10:40:58 +0000

Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
number of tests, all because of the codecs/_multibytecodecs issue described
here (it's not a Py3K issue, just something Py3K discovers):
http://mail.python.org/pipermail/python-dev/2006-April/064051.html

Hye-Shik Chang promised to look for a fix, so no need to fix it here. The
tests that are expected to break are:

test_codecencodings_cn
test_codecencodings_hk
test_codecencodings_jp
test_codecencodings_kr
test_codecencodings_tw
test_codecs
test_multibytecodec

This merge fixes an actual test failure (test_weakref) in this branch,
though, so I believe merging is the right thing to do anyway.

Dummy-merge 2.6 branch into 2.7 branch.

Sat, 05 Mar 2011 15:15:07 +0100

Dummy-merge 2.6 branch into 2.7 branch.

Untabify C files. Will watch buildbots.

Sun, 09 May 2010 14:46:46 +0000

Untabify C files. Will watch buildbots.

http://bugs.python.org/issue6836

Mon, 28 Sep 2009 13:12:38 +0000

http://bugs.python.org/issue6836
The debug memory api now keeps track of which external API (PyMem_* or PyObject_*) was used to allocate each block and treats any API violation as an error. Added separate _PyMem_DebugMalloc functions for the Py_Mem API instead of having it use the _PyObject_DebugMalloc functions.

Issue #4688: Add a heuristic so that tuples and dicts containing only

Mon, 23 Mar 2009 18:41:45 +0000

Issue #4688: Add a heuristic so that tuples and dicts containing only
untrackable objects are not tracked by the garbage collector. This can
reduce the size of collections and therefore the garbage collection overhead
on long-running programs, depending on their particular use of datatypes.

(trivia: this makes the "binary_trees" benchmark from the Computer Language
Shootout 40% faster)

Recorded merge of revisions 81029 via svnmerge from

Sun, 09 May 2010 15:15:40 +0000

#1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.

Wed, 19 Dec 2007 02:37:44 +0000

#1629: Renamed Py_Size, Py_Type and Py_Refcnt to Py_SIZE, Py_TYPE and Py_REFCNT. Macros for b/w compatibility are available.

PEP 3123: Provide forward compatibility with Python 3.0, while keeping

Sat, 21 Jul 2007 06:55:02 +0000

PEP 3123: Provide forward compatibility with Python 3.0, while keeping
backwards compatibility. Add Py_Refcnt, Py_Type, Py_Size, and
PyVarObject_HEAD_INIT.

frame_traverse(): Use the standard Py_VISIT macro.

Sat, 15 Apr 2006 03:22:46 +0000

frame_traverse(): Use the standard Py_VISIT macro.

Py_VISIT: cast the `op` argument to PyObject* when calling
`visit()`. Else the caller has to pay too much attention to
this silly detail (e.g., frame_traverse needs to traverse
`struct _frame *` and `PyCodeObject *` pointers too).

Years in the making.

Sun, 26 Mar 2006 23:27:58 +0000

Years in the making.

objimpl.h, pymem.h: Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build. They're aliases for the
system free() now.

_subprocess.c/sp_handle_dealloc(): Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).

pythonrun.c, tokenizer.c, parsermodule.c: I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject. Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.

If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.

Make PyGC_Collect() use Py_ssize_t.

Sat, 04 Mar 2006 20:01:53 +0000

Make PyGC_Collect() use Py_ssize_t.

Change GC refcount to Py_ssize_t.

Wed, 01 Mar 2006 16:56:25 +0000

Change GC refcount to Py_ssize_t.

Change _PyObject_GC_Resize to expect Py_ssize_t.

Thu, 16 Feb 2006 14:56:14 +0000

Change _PyObject_GC_Resize to expect Py_ssize_t.

Merge ssize_t branch.

Wed, 15 Feb 2006 17:27:45 +0000

Merge ssize_t branch.

Fix PyGC_Collect() to be exported from the built DLL on Windows. (Fix given

Thu, 26 Jan 2006 18:59:06 +0000

Fix PyGC_Collect() to be exported from the built DLL on Windows. (Fix given
by Matt Messier).

[ 1243081 ] repair typos

Fri, 22 Jul 2005 18:40:02 +0000

[ 1243081 ] repair typos

Formalize that the Py_VISIT macro requires that the tp_traverse

Thu, 15 Jul 2004 04:05:59 +0000

Formalize that the Py_VISIT macro requires that the tp_traverse
implementation it's used in must give its arguments specific names.

Documented the new Py_VISIT macro to simplify implementation of

Wed, 14 Jul 2004 19:08:17 +0000

Documented the new Py_VISIT macro to simplify implementation of
tp_traverse handlers. (Tim made me do it. ;)

- New C API PyGC_Collect(), same as calling gc.collect().

Thu, 17 Apr 2003 17:29:22 +0000

- New C API PyGC_Collect(), same as calling gc.collect().
- Call this in Py_Finalize().
- Expand the Misc/NEWS text on PY_LONG_LONG.

Excise DL_EXPORT from Include.

Mon, 12 Aug 2002 07:21:58 +0000

Excise DL_EXPORT from Include.

Thanks to Skip Montanaro and Kalle Svensson for the patches.

Removed WITH_CYCLE_GC #ifdef-ery. Holes:

Sun, 07 Jul 2002 03:59:34 +0000

Removed WITH_CYCLE_GC #ifdef-ery. Holes:

+ I'm not sure what to do about configure.in. Left it alone.

+ Ditto pyexpat.c. Fred or Martin will know what to do.

Finished transitioning to using gc_refs to track gc objects' states.

Tue, 02 Jul 2002 18:12:35 +0000

Finished transitioning to using gc_refs to track gc objects' states.
This was mostly a matter of adding comments and light code rearrangement.
Upon untracking, gc_next is still set to NULL. It's a cheap way to
provoke memory faults if calling code is insane. It's also used in some
way by the trashcan mechanism.

Reserved another gc_refs value for untracked objects. Every live gc

Tue, 02 Jul 2002 00:52:30 +0000

Reserved another gc_refs value for untracked objects. Every live gc
object should now have a well-defined gc_refs value, with clear transitions
among gc_refs states. As a result, none of the visit_XYZ traversal
callbacks need to check IS_TRACKED() anymore, and those tests were removed.
(They were already looking for objects with specific gc_refs states, and
the gc_refs state of an untracked object can no longer match any other
gc_refs state by accident.)
Added more asserts.
I expect that the gc_next == NULL indicator for an untracked object is
now redundant and can also be removed, but I ran out of time for this.

_PyGC_generation0 is now a pointer

Sat, 04 May 2002 05:36:06 +0000

_PyGC_generation0 is now a pointer

Moving pymalloc along.

Sun, 28 Apr 2002 04:11:46 +0000

Moving pymalloc along.

As threatened, PyMem_{Free, FREE} also invoke the object deallocator now
when pymalloc is enabled (well, it does when pymalloc isn't enabled too,
but in that case "the object deallocator" is plain free()).

This is maximally backward-compatible, but it leaves a bitter aftertaste.

Also massive reworking of comments.

_PyObject_DebugDumpStats: renamed to _PyObject_DebugMallocStats.

Sat, 13 Apr 2002 08:29:14 +0000

_PyObject_DebugDumpStats: renamed to _PyObject_DebugMallocStats.
Added code to call this when PYMALLOC_DEBUG is enabled, and envar
PYTHONMALLOCSTATS is set, whenever a new arena is obtained and once
late in the Python shutdown process.

First cut at repairing out-of-date comments; make alignment of macro defs

Fri, 12 Apr 2002 05:21:34 +0000

First cut at repairing out-of-date comments; make alignment of macro defs
all the same within the #ifdef WITH_PYMALLOC block.

Remove PyMalloc_* symbols. PyObject_Malloc now uses pymalloc if

Fri, 12 Apr 2002 02:38:45 +0000

Remove PyMalloc_* symbols. PyObject_Malloc now uses pymalloc if
it's enabled.

Allow PyObject_Del, PyObject_Free, and PyObject_GC_Del to be used as
function designators. Provide source compatibility macros.

Make PyObject_GC_Track and PyObject_GC_UnTrack functions instead of
trivial macros wrapping functions.

Add _Py_AS_GC macro. It will be used by the trashcan code on object.c.

Thu, 28 Mar 2002 21:06:16 +0000

Add _Py_AS_GC macro. It will be used by the trashcan code on object.c.

Make PyObject_{NEW,New,Del,DEL} always use the standard malloc (PyMem_*)

Fri, 22 Mar 2002 15:25:18 +0000

Make PyObject_{NEW,New,Del,DEL} always use the standard malloc (PyMem_*)
and not pymalloc. Add the functions PyMalloc_New, PyMalloc_NewVar, and
PyMalloc_Del that will use pymalloc if it's enabled. If pymalloc is
not enabled then they use the standard malloc (PyMem_*).

Drop the PyCore_* memory API.

Mon, 18 Mar 2002 21:06:21 +0000

Drop the PyCore_* memory API.

Simpilify PyCore_* macros by assuming the function prototypes for

Mon, 18 Mar 2002 18:13:41 +0000

Simpilify PyCore_* macros by assuming the function prototypes for
malloc() and free() don't change.

PyGC_Head: Use "long double" instead of "double" as the worst-case

Thu, 28 Feb 2002 19:38:51 +0000

PyGC_Head: Use "long double" instead of "double" as the worst-case
alignment gimmick. David Abrahams notes that the standard "long double"
actually requires stricter alignment than "double" on some Tru64 box.
On my box and yours , it's the same, so no harm done on most
boxes.

SF bug [#467145] Python 2.2a4 build problem on HPUX 11.0.

Thu, 11 Oct 2001 18:31:31 +0000

SF bug [#467145] Python 2.2a4 build problem on HPUX 11.0.
The platform requires 8-byte alignment for doubles, but the GC header
was 12 bytes and that threw off the natural alignment of the double
members of a subtype of complex. The fix puts the GC header into a
union with a double as the other member, to force no-looser-than
double alignment of GC headers. On boxes that require 8-byte alignment
for doubles, this may add pad bytes to the GC header accordingly; ditto
for platforms that *prefer* 8-byte alignment for doubles. On platforms
that don't care, it shouldn't change the memory layout (because the
size of the old GC header is certainly greater than the size of a double
on all platforms, so unioning with a double shouldn't change size or
alignment on such boxes).

Guido suggests, and I agree, to insist that SIZEOF_VOID_P be a power of 2.

Sun, 07 Oct 2001 03:54:51 +0000

Guido suggests, and I agree, to insist that SIZEOF_VOID_P be a power of 2.
This simplifies the rounding in _PyObject_VAR_SIZE, allows to restore the
pre-rounding calling sequence, and allows some nice little simplifications
in its callers. I'm still making it return a size_t, though.

_PyObject_VAR_SIZE: always round up to a multiple-of-pointer-size value.

Sat, 06 Oct 2001 21:27:34 +0000

_PyObject_VAR_SIZE: always round up to a multiple-of-pointer-size value.
As Guido suggested, this makes the new subclassing code substantially
simpler. But the mechanics of doing it w/ C macro semantics are a mess,
and _PyObject_VAR_SIZE has a new calling sequence now.

Question: The PyObject_NEW_VAR macro appears to be part of the public API.
Regardless of what it expands to, the notion that it has to round up the
memory it allocates is new, and extensions containing the old
PyObject_NEW_VAR macro expansion (which was embedded in the
PyObject_NEW_VAR expansion) won't do this rounding. But the rounding
isn't actually *needed* except for new-style instances with dict pointers
after a variable-length blob of embedded data. So my guess is that we do
not need to bump the API version for this (as the rounding isn't needed
for anything an extension can do unless it's recompiled anyway). What's
your guess?

Repaired the debug Windows deaths in test_descr, by allocating enough

Sat, 06 Oct 2001 19:04:01 +0000

Repaired the debug Windows deaths in test_descr, by allocating enough
pad memory to properly align the __dict__ pointer in all cases.

gcmodule.c/objimpl.h, _PyObject_GC_Malloc:
+ Added a "padding" argument so that this flavor of malloc can allocate
enough bytes for alignment padding (it can't know this is needed, but
its callers do).

typeobject.c, PyType_GenericAlloc:
+ Allocated enough bytes to align the __dict__ pointer.
+ Sped and simplified the round-up-to-PTRSIZE logic.
+ Added blank lines so I could parse the if/else blocks <0.7 wink>.

Add Garbage Collection support to new-style classes (not yet to their

Tue, 02 Oct 2001 21:24:57 +0000

Add Garbage Collection support to new-style classes (not yet to their
instances).

Also added GC support to various auxiliary types: super, property,
descriptors, wrappers, dictproxy. (Only type objects have a tp_clear
field; the other types are.)

One change was necessary to the GC infrastructure. We have statically
allocated type objects that don't have a GC header (and can't easily
be given one) and heap-allocated type objects that do have a GC
header. Giving these different metatypes would be really ugly: I
tried, and I had to modify pickle.py, cPickle.c, copy.py, add a new
invent a new name for the new metatype and make it a built-in, change
affected tests... In short, a mess. So instead, we add a new type
slot tp_is_gc, which is a simple Boolean function that determines
whether a particular instance has GC headers or not. This slot is
only relevant for types that have the (new) GC flag bit set. If the
tp_is_gc slot is NULL (by far the most common case), all instances of
the type are deemed to have GC headers. This slot is called by the
PyObject_IS_GC() macro (which is only used twice, both times in
gcmodule.c).

I also changed the extern declarations for a bunch of GC-related
functions (_PyObject_GC_Del etc.): these always exist but objimpl.h
only declared them when WITH_CYCLE_GC was defined, but I needed to be
able to reference them without #ifdefs. (When WITH_CYCLE_GC is not
defined, they do the same as their non-GC counterparts anyway.)