on_exit callback not called when a Process exits

[ggreg]

on_exit callback is not called when a Process exits.

Consequently the semaphore of a Pool is not released.

[ggreg]

Introduction

There are at least three issues with the execution of on_exit.

When the child process exits, its parent receives a SIGCHLD signal that is handled by Process.on_sigchld(). The three cases below show how the handler may not be called or able to execute the on_exit callback.

Interleaved calls

See #6.

Parent exits

If the parent process exits before the child process, the SIGCHLD signal handler that calls the function will never be called.

Several processes with on_exit

See #7.

Alternatives

The first alternative I'm considering is atexit. It allows to register functions to call when a process exits.

Another one would call select() or poll() on a file descriptor that the parent and child processes share (see http://stackoverflow.com/a/8976461/1059876). As this solution adds complexity, I prefer considering it only if the former does not work.

[ggreg]

An atexit callback would not work because the child process exited with os._exit(). If we replace os._exit() by sys.exit(), it would then raise a SystemExit error.

[ggreg]

Different places to execute the callback

There should be parameters with different semantics depending on where the callback back is executing.

Inside the child process

Normal Child Termination

The callback is executed after the child process's target finishes its execution. There is no real need for this because the caller can wrap the original target with a decorator and customize its behavior.

Whenever the child exits

When could the child process's target not execute an exit callback? When it is interrupted by a signal. Then either:

1. It can handle the signal, such as SIG_TERM and we have to install a signal handler for every signal we want to handle.
2. Either it cannot, for example if it receives a SIG_KILL signal. In this case, we have to find a way to execute the callback in the parent process.

Inside the parent process

The callback will execute in the process that is calling the child whenever the child process exits. This callback will not be executed if the parent process no longer exists.

For now, I only see a SIG_CHLD handler as a solution.

How to avoid confusion

The user should be of a necessary trade-off, either the callback requires that the parent process is still alive to execute it, either it does not. But in this case, nothing ensures the callback will ever be executed.

Let's take the example of releasing a semaphore. An object bounds some resources with a semaphore and needs to release the semaphore when a child process exits. Does it make sense to try to ensure the child process releases the semaphore when it exits even if the parent process is no longer there? No because, the semaphore is needed by the parent. So, if it is not here, it does not care if the semaphore is consistent.

Bottom Line

We need two kind of parameters:

• One that binds signal handlers in the child process.
• Another that defines a SIG_CHLD handler in the parent.

Beware that the SIG_CHLD handler may be defined by different Process objects. See the problem documented in a comment above: If two process A and B define an on_exit callback, B will override the behaviour of A because it overrides the SIGCHLD signal handler of A with its one. We can solve this by defining a protocol to stack signal handlers we are registering. This could be providing in a separate module.

[ggreg]

@oleiade what is your take on the comment above?

[ggreg]

Now we have to address the shortcomings documented in #5 (comment):

• Process.on_sigchld() is never called (in Interleaved Calls) Process.on_sigchld do not work when Process.join was called #6.
• Several processes with on_exit Signal handler for a Process overrides all previous ones #7.