Threads and fork (): Think twice before mixing them.

Address: http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

 

When debugging a program I Came authentication ss a bug that was caused by using
Fork (2) in a multi-threaded program. I thought it's worth to write some words about mixing POSIX Threads

Fork (2) because there are non-obvious problems when doing that.

What happens after fork () in a multi-threadeed Program

The
Fork (2) function creates a copy of the process, all memory pages are copied, open file descriptors are copied etc. all this stuff is intuitive for a Unix programmer. one important thing that differs the child process from the parent is that the child has
Only one thread. cloning the whole process with all threads wocould be problematic and in most cases not what the programmer wants. just think about it: What to do with threads that are susponded executing a system call? So

Fork (2) Call clones just the thread which executed it.

What are the problems

 

Critical sections, mutexes

The non-obvious problem in this approach is that at the moment of
Fork (2) call some threads may be in critical sections of code, doing non-atomic operations protected by mutexes. in the child process the threads just disappears and left data half-modified without any possibility to "fix" them, there is no way to say what
Other threads were doing and what shoshould be done to make the data consistent. Moreover: State of mutexes is undefined, they might be unusable and the only way to use them in the child is to call
pthread_mutex_init()To reset them to a usable state. It's implementation dependent how mutexes behave after

Fork (2) was called. On my Linux machine locked mutexes are locked in the child.

Library functions

Problem with mutexes and critical code sections implies another non-obvious issue. it's theoretically possible to write your code executed in threads so that you are sure it's safe to call fork when such threads run but in practice there is one big problem:
Library functions. you're never sure if a library function you are using doesn' t use global data. even if it is thread safe, it may be achieved using mutexes internally. you are never sure. even system library functions that are thread-safe may use locks internally.
One non-obvious example ismalloc()Function which at least in multi-threaded programs on my system uses locks. So it's not safe
Call
Fork (2) at the moment some other thread CILSmalloc()! What the standard says about it? After

Fork (2) in a multi-threaded program you may only call async-safe functions (listed in

Signal (7). It's similar limitation to the list of functions you are allowed to call in

Signal handler and the reason is similar: in both cases the thread might be "interrupted" while executing a function.

Here is a list of few functions that use locks internally on my system, just to show you that really almost nothing is safe:

• malloc()
• Stdio functions likeprintf()-This is required by the standard.
• syslog()

Execve () and open file descriptors

It seems that calling
Execve (2) to start another program is the only sane reason you wowould like to call

Fork (2) in a multi-threaded program. But even doing that has at least one problem. When calling

Execve (2) one must remember that open file descriptors remain open and the program that was executed may read and write to them. it creates a problem if you leave open file descriptor at the time you call

Execve (2) that was not intended to be visible by the executed program. It even creates security issues in some cases. There is a solution for that: you must set
FD_CLOEXECFlag on all file descriptors using

Fcntl (2) So they are automatically closed on new programs execution. Unfortunately it's not as simple in multi-threaded program. When using

Fcntl (2) to setFD_CLOEXECFlag there is a race:

 
 

  
  
fd = open ("file", O_RDWR | O_CREAT | O_TRUNC, 0600);
  
  
if (fd < 0) {
  
  
 perror ("open()");
  
  
 return 0;
  
  
}
  
  
 
  
  
fcntl (fd, F_SETFD, FD_CLOEXEC);
 
 

When another thread executes
Fork (2) and
Execve (2) Just between this thread does
Open (2) but before
Fcntl (2) a new program is started having this file descriptor duplicated. this is not what we want. A solution was created with newer standards (like POSIX.1-2008) and newer Linux kernel (changes in 2.6.23 and later ). we now have
O_CLOEXECFlag to

Open (2) function, so the whole operation of opening a file and setting
FD_CLOEXECFlag is atomic.

There are other ways to create file descriptors than using
Open (2): duplicating them
DUP (2), creating sockets
Socket (2) etc. All those functions have now a flag similar
O_CLOEXECOr a newer version that can take similar flag (some of them, like

Dup2 (2) does not have a flags argument, so
Dup3 (2) was created ).

It's worth to mention that a similar thing may happen in a single threaded program when it does

Fork (2) and
Execve (2) in a signal handler. this operation is perfectly legal because both of the funnal are async-safe and can be called from a signal handler, but the problem is the program may be interrupted

Open (2) and
Fcntl (2 ).

For more information about the new API to setFD_CLOEXECFlag see

Ulrich drepper's blog: secure file descriptor handling.

Useful system functions: pthread_atfork ()

One useful function that tries to solve the problem
Fork (2) in multi-threaded programs is
Pthread_atfork (). It has the following prototype:

 
 

  
  
int pthread_atfork(void (*prepare)(void), void (*parent)(void), void (*child)(void));
 
 

It allows to set handler functions that will be automatically executed on fork call:

• Prepare-called just before a new process is created.
• Parent-called after a new process is created in the parent.
• Child-called after a new process is created in the child.

The purpose of this call is to deal with critical sections of a multi-threaded program at the time

Fork (2) is called. A typical scenario is when in the prepare handler mutexes are locked, in the parent handler unlocked and in the child handler reinitialized.

Summary

In my opinion there are so many problems
Fork (2) in multi-threaded programs that it's almost impossible to do it right. The only clear case is to call

Execve (2) in the child process just after
Fork (2). If you want to do something more, just do it some other way, really. From my experience it's not worth trying to make

Fork (2) Call save, evenpthread_atfork(). I truly hope you read this article before hitting problems described here.

Resources

 

• Fork () description from the Open Group Single UNIX Specification
• Ulrich drepper's blog: secure file descriptor handling
• Pthread_atfork ()