Running long background processes with double fork magic

Note: In the following post child process is the process created after first fork() and grand child process is the process created after second fork().   

In one of my projects recently I had to run a shell command as a background process. The system was performance intensive, so I had to run a child process and continue execution on the parent process.

           We were using Python on Ubuntu so initially it seemed very easy. I thought of just using the subprocess library available for python. I started a background process with subprocess.Popen() and not call a Popen.wait() for it because I had to run the shell command in the background.

PROBLEM:

           Initially I thought it worked but when I checked the process tree. I noticed that the child processes completed  and entered a <defunct> state were not releasing memory. 

SOLUTION:

          After some searching I came across this post and come to know about the double fork magic.

So what is a double fork magic? A double fork is something of a hack to run child processes and at the same time continue execution on the parent process.

In a normal situation where we want to execute a command or logic, we expect to run for a short time, we would do something like this:

                    ------------------------
                    |    Parent Process   |
                    ------------------------
                                 |

                              fork()
                                 |

                                 |---------------------------------
                    -----------------------                           |

                    |   Parent Process   |                 ---------------------

                    -----------------------                 |   Child Process  |

                                                                    ---------------------

In this scenario, we create a child process and wait for the child process to complete its execution by calling wait() in the Parent process. As soon as the child is forked it starts execution and parent process enters a blocking state and halts its execution until the child process signals completion. When the child is done, it sends a signal to the parent process and then the parent process continues execution.

Now a double fork magic looks like this:

                    ------------------------
                    |    Parent Process   |
                    ------------------------
                                 |

                              fork()
                                 |

                                 ----------------------------------
                    -----------------------                           |

                    |   Parent Process   |              ---------------------

                    -----------------------              |   Child Process  |

                                                                 --------------------- 

                                                                             |

                                                                          fork()

                                                                             |

                                                           ------------------------------                           

                                                           |   Grand Child Process  |                                                                                                                           ------------------------------ 

In this scenario we fork a child and in that child we don't execute the command or logic but fork another process, a grand child process and execute the command/logic in the grand child process. But in this case we don't call a wait() on neither the parent process nor the child process. Now why this works, is why its called *magic*.

The child process creates the grand child and exits as its sole purpose is to create a grandchild. As the child process exits, it calls an automatic join() which is propagated to all its children, (in our case, grand child process). So when the grand child completes its execution, it doesn't enters a <defunct> state as it has already received a join(). Similarly the child exits after a very short time and returns as soon as it has created the grandchild and doesn't enter a <defunct> state.   

SUMMARY:

         When you want to run a command in the background you expect to run for a short time, use wait() until the child returns. But when you expect to run the child for a much longer time, the double fork magic comes in handy.

Here is an example of a Daemon class in python implemented through the double fork magic.