cpython-lru_cache-weakref: Lib/multiprocessing/util.py annotate

annotate Lib/multiprocessing/util.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	3e397ecedc55
children

rev	line source
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	1 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	2 # Module providing various facilities to other parts of the package
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	3 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	4 # multiprocessing/util.py
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	5 #
66885 b084b50e4aa6 9162: fix license in multiprocessing files R. David Murray <rdmurray@bitdance.com> parents: 51024 diff changeset	6 # Copyright (c) 2006-2008, R Oudkerk
76663 25cc0c8fd4e2 Mark multiprocessing files with "Licensed to PSF under a Contributor Agreement" Richard Oudkerk <shibturn@gmail.com> parents: 73513 diff changeset	7 # Licensed to PSF under a Contributor Agreement.
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	8 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	9
77035 347735ec92eb #12098: Make multiprocessing's child processes inherit sys.flags on Windows Richard Oudkerk <shibturn@gmail.com> parents: 76863 diff changeset	10 import sys
70690 568a3ba088e4 Issue #12040: Expose a new attribute `sentinel` on instances of Antoine Pitrou <solipsis@pitrou.net> parents: 66885 diff changeset	11 import functools
77141 59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	12 import os
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	13 import itertools
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	14 import weakref
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	15 import atexit
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	16 import threading # we want threading to install it's
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	17 # cleanup function before multiprocessing does
77036 2034b3de1144 Move private function _args_from_interpreter_flags() to subprocess.py, so Antoine Pitrou <solipsis@pitrou.net> parents: 77035 diff changeset	18 from subprocess import _args_from_interpreter_flags
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	19
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	20 from . import process
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	21
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	22 __all__ = [
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	23 'sub_debug', 'debug', 'info', 'sub_warning', 'get_logger',
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	24 'log_to_stderr', 'get_temp_dir', 'register_after_fork',
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	25 'is_exiting', 'Finalize', 'ForkAwareThreadLock', 'ForkAwareLocal',
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	26 'close_all_fds_except', 'SUBDEBUG', 'SUBWARNING',
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	27 ]
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	28
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	29 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	30 # Logging
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	31 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	32
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	33 NOTSET = 0
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	34 SUBDEBUG = 5
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	35 DEBUG = 10
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	36 INFO = 20
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	37 SUBWARNING = 25
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	38
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	39 LOGGER_NAME = 'multiprocessing'
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	40 DEFAULT_LOGGING_FORMAT = '[%(levelname)s/%(processName)s] %(message)s'
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	41
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	42 _logger = None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	43 _log_to_stderr = False
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	44
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	45 def sub_debug(msg, *args):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	46 if _logger:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	47 _logger.log(SUBDEBUG, msg, *args)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	48
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	49 def debug(msg, *args):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	50 if _logger:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	51 _logger.log(DEBUG, msg, *args)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	52
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	53 def info(msg, *args):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	54 if _logger:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	55 _logger.log(INFO, msg, *args)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	56
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	57 def sub_warning(msg, *args):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	58 if _logger:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	59 _logger.log(SUBWARNING, msg, *args)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	60
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	61 def get_logger():
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	62 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	63 Returns logger used by multiprocessing
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	64 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	65 global _logger
73513 60fd58a56f44 Remove unused or redundant imports in concurrent.futures and multiprocessing. Florent Xicluna <florent.xicluna@gmail.com> parents: 73085 diff changeset	66 import logging
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	67
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	68 logging._acquireLock()
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	69 try:
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	70 if not _logger:
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	71
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	72 _logger = logging.getLogger(LOGGER_NAME)
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	73 _logger.propagate = 0
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	74
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	75 # XXX multiprocessing should cleanup before logging
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	76 if hasattr(atexit, 'unregister'):
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	77 atexit.unregister(_exit_function)
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	78 atexit.register(_exit_function)
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	79 else:
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	80 atexit._exithandlers.remove((_exit_function, (), {}))
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	81 atexit._exithandlers.append((_exit_function, (), {}))
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	82
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	83 finally:
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	84 logging._releaseLock()
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	85
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	86 return _logger
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	87
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	88 def log_to_stderr(level=None):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	89 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	90 Turn on logging and add a handler which prints to stderr
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	91 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	92 global _log_to_stderr
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	93 import logging
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	94
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	95 logger = get_logger()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	96 formatter = logging.Formatter(DEFAULT_LOGGING_FORMAT)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	97 handler = logging.StreamHandler()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	98 handler.setFormatter(formatter)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	99 logger.addHandler(handler)
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	100
4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	101 if level:
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	102 logger.setLevel(level)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	103 _log_to_stderr = True
51024 4e8f28b3ea0a merge r68915 to py3k Jesse Noller <jnoller@gmail.com> parents: 50883 diff changeset	104 return _logger
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	105
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	106 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	107 # Function returning a temp directory which will be removed on exit
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	108 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	109
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	110 def get_temp_dir():
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	111 # get name of a temp directory which will be automatically cleaned up
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	112 tempdir = process.current_process()._config.get('tempdir')
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	113 if tempdir is None:
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	114 import shutil, tempfile
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	115 tempdir = tempfile.mkdtemp(prefix='pymp-')
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	116 info('created temp directory %s', tempdir)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	117 Finalize(None, shutil.rmtree, args=[tempdir], exitpriority=-100)
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	118 process.current_process()._config['tempdir'] = tempdir
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	119 return tempdir
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	120
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	121 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	122 # Support for reinitialization of objects when bootstrapping a child process
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	123 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	124
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	125 _afterfork_registry = weakref.WeakValueDictionary()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	126 _afterfork_counter = itertools.count()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	127
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	128 def _run_after_forkers():
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	129 items = list(_afterfork_registry.items())
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	130 items.sort()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	131 for (index, ident, func), obj in items:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	132 try:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	133 func(obj)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	134 except Exception as e:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	135 info('after forker raised exception %s', e)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	136
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	137 def register_after_fork(obj, func):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	138 _afterfork_registry[(next(_afterfork_counter), id(obj), func)] = obj
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	139
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	140 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	141 # Finalization using weakrefs
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	142 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	143
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	144 _finalizer_registry = {}
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	145 _finalizer_counter = itertools.count()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	146
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	147
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	148 class Finalize(object):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	149 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	150 Class which supports object finalization using weakrefs
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	151 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	152 def __init__(self, obj, callback, args=(), kwargs=None, exitpriority=None):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	153 assert exitpriority is None or type(exitpriority) is int
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	154
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	155 if obj is not None:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	156 self._weakref = weakref.ref(obj, self)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	157 else:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	158 assert exitpriority is not None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	159
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	160 self._callback = callback
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	161 self._args = args
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	162 self._kwargs = kwargs or {}
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	163 self._key = (exitpriority, next(_finalizer_counter))
77141 59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	164 self._pid = os.getpid()
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	165
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	166 _finalizer_registry[self._key] = self
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	167
71267 27224f7ef000 Rebind locally the globals which can be looked up at shutdown Antoine Pitrou <solipsis@pitrou.net> parents: 70691 diff changeset	168 def __call__(self, wr=None,
27224f7ef000 Rebind locally the globals which can be looked up at shutdown Antoine Pitrou <solipsis@pitrou.net> parents: 70691 diff changeset	169 # Need to bind these locally because the globals can have
27224f7ef000 Rebind locally the globals which can be looked up at shutdown Antoine Pitrou <solipsis@pitrou.net> parents: 70691 diff changeset	170 # been cleared at shutdown
27224f7ef000 Rebind locally the globals which can be looked up at shutdown Antoine Pitrou <solipsis@pitrou.net> parents: 70691 diff changeset	171 _finalizer_registry=_finalizer_registry,
77347 9257ea91df3d Make Finalize reserve a reference to os.getpid in case called at shutdown Richard Oudkerk <shibturn@gmail.com> parents: 77141 diff changeset	172 sub_debug=sub_debug, getpid=os.getpid):
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	173 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	174 Run the callback unless it has already been called or cancelled
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	175 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	176 try:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	177 del _finalizer_registry[self._key]
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	178 except KeyError:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	179 sub_debug('finalizer no longer registered')
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	180 else:
77347 9257ea91df3d Make Finalize reserve a reference to os.getpid in case called at shutdown Richard Oudkerk <shibturn@gmail.com> parents: 77141 diff changeset	181 if self._pid != getpid():
77141 59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	182 sub_debug('finalizer ignored because different process')
59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	183 res = None
59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	184 else:
59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	185 sub_debug('finalizer calling %s with args %s and kwargs %s',
59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	186 self._callback, self._args, self._kwargs)
59567c117b0e Issue #14548: Make multiprocessing finalizers check pid before running Richard Oudkerk <shibturn@gmail.com> parents: 77037 diff changeset	187 res = self._callback(self._args, *self._kwargs)
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	188 self._weakref = self._callback = self._args = \
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	189 self._kwargs = self._key = None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	190 return res
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	191
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	192 def cancel(self):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	193 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	194 Cancel finalization of the object
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	195 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	196 try:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	197 del _finalizer_registry[self._key]
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	198 except KeyError:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	199 pass
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	200 else:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	201 self._weakref = self._callback = self._args = \
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	202 self._kwargs = self._key = None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	203
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	204 def still_active(self):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	205 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	206 Return whether this finalizer is still waiting to invoke callback
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	207 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	208 return self._key in _finalizer_registry
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	209
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	210 def __repr__(self):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	211 try:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	212 obj = self._weakref()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	213 except (AttributeError, TypeError):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	214 obj = None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	215
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	216 if obj is None:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	217 return '<Finalize object, dead>'
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	218
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	219 x = '<Finalize object, callback=%s' % \
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	220 getattr(self._callback, '__name__', self._callback)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	221 if self._args:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	222 x += ', args=' + str(self._args)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	223 if self._kwargs:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	224 x += ', kwargs=' + str(self._kwargs)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	225 if self._key[0] is not None:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	226 x += ', exitprority=' + str(self._key[0])
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	227 return x + '>'
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	228
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	229
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	230 def _run_finalizers(minpriority=None):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	231 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	232 Run all finalizers whose exit priority is not None and at least minpriority
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	233
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	234 Finalizers with highest priority are called first; finalizers with
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	235 the same priority will be called in reverse order of creation.
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	236 '''
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	237 if _finalizer_registry is None:
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	238 # This function may be called after this module's globals are
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	239 # destroyed. See the _exit_function function in this module for more
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	240 # notes.
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	241 return
78924 89a5169f2979 Fixed whitespace Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 78922 diff changeset	242
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	243 if minpriority is None:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	244 f = lambda p : p[0][0] is not None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	245 else:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	246 f = lambda p : p[0][0] is not None and p[0][0] >= minpriority
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	247
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	248 items = [x for x in list(_finalizer_registry.items()) if f(x)]
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	249 items.sort(reverse=True)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	250
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	251 for key, finalizer in items:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	252 sub_debug('calling %s', finalizer)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	253 try:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	254 finalizer()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	255 except Exception:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	256 import traceback
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	257 traceback.print_exc()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	258
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	259 if minpriority is None:
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	260 _finalizer_registry.clear()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	261
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	262 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	263 # Clean up on exit
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	264 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	265
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	266 def is_exiting():
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	267 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	268 Returns true if the process is shutting down
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	269 '''
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	270 return _exiting or _exiting is None
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	271
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	272 _exiting = False
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	273
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	274 def _exit_function(info=info, debug=debug, _run_finalizers=_run_finalizers,
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	275 active_children=process.active_children,
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	276 current_process=process.current_process):
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	277 # We hold on to references to functions in the arglist due to the
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	278 # situation described below, where this function is called after this
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	279 # module's globals are destroyed.
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	280
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	281 global _exiting
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	282
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	283 if not _exiting:
78926 db67b848ddc3 Issue #15881: Fixed 3.2 backport. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 78924 diff changeset	284 _exiting = True
db67b848ddc3 Issue #15881: Fixed 3.2 backport. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 78924 diff changeset	285
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	286 info('process shutting down')
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	287 debug('running all "atexit" finalizers with priority >= 0')
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	288 _run_finalizers(0)
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	289
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	290 if current_process() is not None:
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	291 # We check if the current process is None here because if
80929 15a391919deb Issue #16714: use 'raise' exceptions, don't 'throw'. Andrew Svetlov <andrew.svetlov@gmail.com> parents: 79019 80928 diff changeset	292 # it's None, any call to ``active_children()`` will raise
79019 2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	293 # an AttributeError (active_children winds up trying to
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	294 # get attributes from util._current_process). One
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	295 # situation where this can happen is if someone has
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	296 # manipulated sys.modules, causing this module to be
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	297 # garbage collected. The destructor for the module type
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	298 # then replaces all values in the module dict with None.
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	299 # For instance, after setuptools runs a test it replaces
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	300 # sys.modules with a copy created earlier. See issues
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	301 # #9775 and #15881. Also related: #4106, #9205, and
2b79b4848f44 Issue #15881: Clarify comment in exit function Richard Oudkerk <shibturn@gmail.com> parents: 78928 diff changeset	302 # #9207.
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	303
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	304 for p in active_children():
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	305 if p.daemon:
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	306 info('calling terminate() for daemon %s', p.name)
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	307 p._popen.terminate()
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	308
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	309 for p in active_children():
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	310 info('calling join() for process %s', p.name)
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	311 p.join()
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	312
78922 27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	313 debug('running the remaining "atexit" finalizers')
27d410dd5431 Issue #15881: Fixed atexit hook in multiprocessing. Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66885 diff changeset	314 _run_finalizers()
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	315
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	316 atexit.register(_exit_function)
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	317
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	318 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	319 # Some fork aware types
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	320 #
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	321
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	322 class ForkAwareThreadLock(object):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	323 def __init__(self):
83428 21314f27a40c Issue #17555: Fix ForkAwareThreadLock so that size of after fork Richard Oudkerk <shibturn@gmail.com> parents: 80929 diff changeset	324 self._reset()
21314f27a40c Issue #17555: Fix ForkAwareThreadLock so that size of after fork Richard Oudkerk <shibturn@gmail.com> parents: 80929 diff changeset	325 register_after_fork(self, ForkAwareThreadLock._reset)
21314f27a40c Issue #17555: Fix ForkAwareThreadLock so that size of after fork Richard Oudkerk <shibturn@gmail.com> parents: 80929 diff changeset	326
21314f27a40c Issue #17555: Fix ForkAwareThreadLock so that size of after fork Richard Oudkerk <shibturn@gmail.com> parents: 80929 diff changeset	327 def _reset(self):
47595 5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	328 self._lock = threading.Lock()
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	329 self.acquire = self._lock.acquire
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	330 self.release = self._lock.release
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	331
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	332 class ForkAwareLocal(threading.local):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	333 def __init__(self):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	334 register_after_fork(self, lambda obj : obj.__dict__.clear())
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	335 def __reduce__(self):
5c6cdfcd6eae Merged revisions 64104,64117 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: diff changeset	336 return type(self), ()
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	337
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	338 #
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	339 # Close fds except those specified
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	340 #
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	341
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	342 try:
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	343 MAXFD = os.sysconf("SC_OPEN_MAX")
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	344 except Exception:
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	345 MAXFD = 256
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	346
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	347 def close_all_fds_except(fds):
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	348 fds = list(fds) + [-1, MAXFD]
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	349 fds.sort()
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	350 assert fds[-1] == MAXFD, 'fd too large'
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	351 for i in range(len(fds) - 1):
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	352 os.closerange(fds[i]+1, fds[i+1])
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	353
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	354 #
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	355 # Start a program with only specified fds kept open
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	356 #
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	357
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	358 def spawnv_passfds(path, args, passfds):
85429 3e397ecedc55 Issue #18865: remove unused import from multiprocessing.util.spawnv_passfds() Victor Stinner <vstinner@wyplay.com> parents: 85427 diff changeset	359 import _posixsubprocess
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	360 passfds = sorted(passfds)
85420 ef889c3d5dc6 Issue #18571: Implementation of the PEP 446: file descriptors and file handles Victor Stinner <victor.stinner@gmail.com> parents: 85169 diff changeset	361 errpipe_read, errpipe_write = os.pipe()
85169 3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	362 try:
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	363 return _posixsubprocess.fork_exec(
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	364 args, [os.fsencode(path)], True, passfds, None, None,
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	365 -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	366 False, False, None)
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	367 finally:
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	368 os.close(errpipe_read)
3b82e0d83bf9 Issue #8713: Support alternative start methods in multiprocessing on Unix. Richard Oudkerk <shibturn@gmail.com> parents: 83428 diff changeset	369 os.close(errpipe_write)

Mercurial > cpython-lru_cache-weakref

annotate Lib/multiprocessing/util.py @ 87690:66c1c9f32567 default tip