Design and implementation of "turn" Linux Futex

Introduction
In compiling the 2.6 kernel, you will see in the compilation options [*] Enable Futex support this item, online search, some data will tell you "do not choose this kernel can not be properly run using GLIBC program", What is Futex? What's the relationship with GLIBC?

1. What is Futex
Futex is the abbreviation for fast userspace mutexes, which was designed by Hubertus Franke, Matthew Kirkwood, Ingo Molnar and Rusty Russell. Several are experts in the field of Linux, which may Ingo Molnar people are more familiar with some, after all, is O (1) Scheduler and the implementation of CFS.

Futex is a fast user space mutex, translated by English. The design idea is not difficult to understand, in traditional UNIX systems, System V IPC (Inter process communication), such as semaphores, Msgqueues, sockets and file lock mechanism (flock ()) The inter-process synchronization mechanism is done for a kernel object operation, which is visible to the processes to be synchronized, providing shared state information and atomic operations. A system call (such as SEMOP ()) must be done in the kernel when the process is to be synchronized. However, the study found that many of the synchronization is non-competitive, that is, a process into the mutex, and then come out of a mutual exclusion zone, often there is no process to enter this mutex or request the same synchronization variable. But in this case, the process also goes into the kernel to see if anyone competes with it, and when it exits, it goes into the kernel to see if there are any processes waiting on the same synchronization variable. These unnecessary system calls (or kernel falls) cause a lot of performance overhead. In order to solve this problem, Futex came into being, Futex is a synchronization mechanism of user-state and kernel-state mixing. First, the synchronized process through mmap share a memory, the Futex variable is in this shared memory and the operation is atomic, when the process attempts to enter the mutex or exit the mutex, the first to see the shared memory of the Futex variable, if no competition occurs, then only modify Futex, and not With the re-execution system call. When the Futex variable is accessed to tell the process that a competition has occurred, the system call is executed to complete the corresponding processing (wait or wake up). Simply put, Futex is through in the user state of the check, (motivation) If you know that there is no competition without falling into the kernel, greatly improving the efficiency of the low-contention time. Linux supports Futex starting from 2.5.7.

2. Futex system call
Futex is a mix of user-and kernel-state, so it takes two parts to work together, and Linux provides Sys_futex system calls to support synchronous processing in the event of a process race.
Its prototype and system call numbers are
#include <linux/futex.h>
#include <sys/time.h>
int Futex (int *uaddr, int op, int val, const struct TIMESPEC *timeout,int *uaddr2, int val3);
#define __NR_FUTEX 240

Although the parameters are a bit long, in fact commonly used is the front three, behind the timeout we can understand, and others are often ignore.
The UADDR is the address of the shared memory under the user state, which is stored in an aligned integer counter.
OP holds the type of operation. There are 5 definitions, here I briefly introduce two kinds, the rest of the interest of their own to man Futex
Futex_wait: atomicity checks whether the value of the counter in Uaddr is Val, and if so, let the process hibernate until the Futex_wake or timeout (time-out). That is, the process is hung to the uaddr corresponding waiting queue.
Futex_wake: Wakes up at most Val waits on the uaddr process.

Visible futex_wait and Futex_wake are only used to suspend or wake up the process, of course, this part of the work can only be done in the kernel state. Some people try to use Futex system calls directly to achieve process synchronization, and hope to get the Futex performance advantage, which is problematic. The Futex synchronization mechanism and Futex system calls should be distinguished. The Futex synchronization mechanism also includes actions under User Configuration, which we will refer to in the next section.

3. Futex synchronization mechanism
All Futex synchronization operations should start with the user space and first create a Futex synchronization variable, which is an integer counter in shared memory.
When the process attempts to hold the lock or to enter the mutex, perform a "down" operation on the Futex, that is, the atomicity of the Futex synchronization variable minus 1. If the synchronization variable becomes 0, no contention occurs and the process executes as usual. If the synchronization variable is a negative number, it means that there is a competition to occur and the futex_wait operation that calls the Futex system call needs to hibernate the current process.
When the process releases the lock or wants to leave the mutex, the Futex is "up", which is atomic to the Futex synchronization variable plus 1. If the synchronization variable is changed from 0 to 1, no competition occurs and the process executes as usual. If the addition of a pre-sync variable is negative, it means that there is a competition to occur, and the futex_wake operation that calls the Futex system call wakes one or more waiting processes.

The atomic plus and minus is usually done with CAS (Compare and Swaps), which is related to the platform. The basic form of CAS is CAS (addr,old,new), which is replaced with new when the value stored in addr equals old. There is a special instruction on the x86 platform to complete it: CMPXCHG.

Visible: Futex is from the user state, by the user state and the nuclear mentality of coordination completed.

4. Input/thread Utilization Futex synchronization
A process or thread can use Futex to synchronize.
For threads, the situation is relatively simple, because the thread shares the virtual memory space, the virtual address can uniquely identify the Futex variable, that is, the thread uses the same virtual address to access the Futex variable.
For processes, the situation is relatively complex because the process has independent virtual memory space, and only through Mmap () lets them share an address space to use the Futex variable. The virtual address that each process uses to access the Futex can be different, as long as the system knows that all of these virtual addresses are mapped to the same physical memory address and uses the physical memory address to uniquely identify the Futex variable.

Summary:
1. Characteristics of the Futex variable: 1) in the shared User Space 2) is a 32-bit integer 3) its operation is atomic
2. Futex can achieve better performance than traditional synchronization mechanisms when the program is low-contention.
3. Do not use Futex system calls directly.
4. The Futex synchronization mechanism can be used for inter-process synchronization or for inter-thread synchronization.

Thread synchronization mechanism in Linux (ii)--in GLIBC

In Linux for multi-threaded development, synchronization is an unavoidable problem. There are three thread synchronization mechanisms defined in the POSIX standard: mutexes (Mutex), Condition Variables (condition variable), and POSIX semaphores (semaphore). Nptl basically implements POSIX, and GLIBC uses NPTL as its own line libraries. So glibc includes the implementation of these three synchronization mechanisms (including, of course, other synchronization mechanisms, such as the read-write lock mentioned in Apue).

Examples of thread synchronization methods commonly used in GLIBC:

Semaphore
Variable definition: sem_t sem;
Initialization: Sem_init (&sem,0,1);
Enter Lock: sem_wait (&sem);
Exit Unlock: Sem_post (&sem);

Mutex
Variable definition: pthread_mutex_t mut;
Initialization: Pthread_mutex_init (&mut,null);
Enter Lock: Pthread_mutex_lock (&mut);
Exit Unlock: Pthread_mutex_unlock (&mut);


What is the relationship between these functions for synchronization and Futex? Let's take a look at the following:
Taking semaphores as an example,
When entering the mutex, the sem_wait (sem_t *sem) is executed, and the sem_wait implementation is as follows:
int sem_wait (sem_t *sem)
{
int *futex = (int *) SEM;
if (atomic_decrement_if_positive (Futex) > 0)
return 0;
int err = lll_futex_wait (Futex, 0);
return-1;
)
The semantics of atomic_decrement_if_positive () is that if an incoming parameter is a positive number, its atomicity is returned immediately. If the semaphore is positive, the semantics of the semaphores means that there is no competition, and if there is no competition, it is returned directly after the semaphore has been reduced by a number.

If the incoming parameter is not a positive number, that means there is a competition, call Lll_futex_wait (futex,0), lll_futex_wait is a macro and expands to:
#define LLL_FUTEX_WAIT (Futex, Val) \
({                                          \
...
__asm __volatile (lll_ebx_load \
Lll_enter_kernel \
Lll_ebx_load \
: "=a" (__status) \
: "0" (Sys_futex), Lll_ebx_reg (Futex), "S" (0), \
"C" (futex_wait), "D" (_val), \
"I" (Offsetof (tcbhead_t, sysinfo)) \
: "Memory"); \
...                                      \
})
You can see that when the competition occurs, sem_wait invokes the Sys_futex system call and executes futex_wait at Val=0, allowing the current thread to hibernate.

As we can see from this example, the Futex is used in the implementation of the semaphores, not only that it uses the Futex system call (again, it is not enough to use the Futex system call again), but that the whole is built on the Futex mechanism, This includes operation under the user state and operation under the kernel mentality. In fact, for other glibc synchronization mechanism is the same, have adopted the Futex as its foundation. So in Futex's manual said: "For most programmers do not need to directly use futexes, instead of relying on the building on the Futex of the system library, such as NPTL line libraries (most programmers would in the fact not being using Futexes directly but instead rely on system libraries built on them, such as the NPTL pthreads implementation). So there will be if you do not enable Futex support when compiling the kernel, you will "not necessarily be able to run the program using GLIBC correctly".

Summary:
1. The thread synchronization methods provided in the glibc, such as the familiar Mutex,semaphore, are mostly constructed on the Futex, except for the special situation, we do not need to implement their own Futex synchronization of the original language.
2. What everyone is going to do, it seems, is what they say in Futex's manual: Correct use of the synchronization methods provided by GLIBC, and in the process of using them, realize that they are done synchronously with the Futex mechanism and Linux.

Thread synchronization mechanism in Linux (iii)--practice

The last talk about thread synchronization in glibc (NPTL), such as Mutex,semaphore, uses Futex as its foundation. So what does the actual use look like and what problems will be encountered?
Let's take a look at an example using Semaphore synchronization.

Sem_t Sem_a;
void *task1 ();

int main (void) {
int ret=0;
pthread_t Thrd1;
Sem_init (&sem_a,0,1);
Ret=pthread_create (&thrd1,null,task1,null); Creating Child Threads
Pthread_join (Thrd1,null); Wait for the child thread to end
}

void *task1 ()
{
int sval = 0;
Sem_wait (&AMP;SEM_A); Amount of signal held
Sleep (5); Do_nothing
Sem_getvalue (&sem_a,&sval);
printf ("Sem value =%d\n", sval);
Sem_post (&AMP;SEM_A); Release semaphore
}

The program is simple, and we create a thread in the main thread (the one that executes main) and wait for it to end with a join. In a child thread, hold the semaphore first, then rest for a while, then release the semaphore and end.
Because there is only one thread in this code that uses semaphores, that is, no thread contention occurs, according to Futex's theory, because there is no competition, all of the lock operations will be done in the user state without executing the system calls into the kernel. We use Strace to track the system calls that occur during the execution of this program:
...
20533 Futex (0xb7db1be8, futex_wait, 20534, NULL <unfinished ...>
20534 Futex (0x8049870, futex_wake, 1) = 0
20533 &lt futex resumed>) = 0
...
20533 is the id,20534 of the main thread, which is the ID of its child thread. To our surprise, this program still happens two times Futex system call, let's analyze what the difference is caused.

1. Unexpected "sem_post ()"
20534 Futex (0x8049870, futex_wake, 1) = 0
The child thread still executes the Futex_wake system call, that is, in Sem_post (&sem_a), when the request kernel wakes up a thread waiting on the sem_a, its return value is 0, indicating that there is no thread waiting at the Sem_a (which is of course, Because just one thread is using sem_a, this time the Futex system calls white. This seems to have something to do with Futex's theory, and let's take a look at the implementation of Sem_post.
int Sem_post (sem_t *sem)
{
int *futex = (int *) SEM;
int nr = Atomic_increment_val (Futex);
int err = Lll_futex_wake (Futex, NR);
return 0;
}
We see that glibc, when implementing Sem_post, adds 1 to Futex atomicity, regardless of the value of Futex, executes Lll_futex_wake (), or Futex (Futex_wake) system call.
In the second part (see previous article), we analyzed the implementation of sem_wait, when there is no competition, there will be no futex call, now it seems to be true, but in sem_post, whether there is no competition, will call Sys_futex (), why this? I think we should combine the semantics of semaphore to understand. In the semantics of Semaphore, sem_wait () means: "Suspend the current process until the value of semaphore is not 0, it will atomically reduce the semaphore count value." "We can see that the semaphore is a blocking or non-blocking thread judged by 0 or non-". That is, no matter how many threads are competing for this lock, the value of Semaphore,semaphore will be 0 if used. This way, when the thread launches the mutex, executes Sem_post (), releases the semaphore, changes its value from 0 to 1, and does not know if the threads are blocked on the semaphore, so it does not matter how the Futex is executed (UADDR, Futex_wake, 1) Try to wake up a process. Conversely, when sem_wait (), if semaphore is changed from 1 to 0, it means that no competition occurs, so there is no need to perform futex system calls. Let's assume that if you throw this semantics aside, if you allow the semaphore value to be negative, you can also implement the Futex mechanism at Sem_post ().

2. "Pthread_join ()" Killed halfway
And how did the other Futex system call caused it? is because Pthread_join ();
In glibc, Pthread_join is also implemented with Futex system calls. Pthread_join in the program (THRD1,NULL); It corresponds to the
20533 Futex (0xb7db1be8, futex_wait, 20534, NULL <unfinished ...>
Well explained that the main thread waits for the child thread (ID number 20534) to end, call Futex (futex_wait), and set the Var parameter to the child thread number (20534) to wait, and then wait on a Futex variable with an address of 0xb7db1be8. When the child thread finishes, the system is responsible for waking the main thread. So the main thread is
20533 &lt futex resumed>) = 0
Resumed running.
It is important to note that if the thread to join is already closed when executing pthread_join (), no more calls to Futex () will block the current process.

3. More competition.
Let's change the above program slightly:
In the main function:
int main (void) {
...
Sem_init (&sem_a,0,1);
Ret=pthread_create (&thrd1,null,task1,null);
Ret=pthread_create (&thrd2,null,task1,null);
Ret=pthread_create (&thrd3,null,task1,null);
Ret=pthread_create (&thrd4,null,task1,null);
Pthread_join (Thrd1,null);
Pthread_join (Thrd2,null);
Pthread_join (Thrd3,null);
Pthread_join (Thrd4,null);
...
}

In this way, more threads are involved in Sem_a's contention. Let's analyze how many times such a program will occur Futex system calls.
1) sem_wait ()
The first incoming thread does not call Futex, and the other threads are called because they want to block, so sem_wait causes 3 Futex (futex_wait) calls.
2) Sem_post ()
All threads will call Futex at Sem_post, resulting in 4 Futex (Futex_wake) calls.
3) Pthread_join ()
Don't forget the Pthread_join (), we are Thread1, Thread2, Thread3, thread4 to join, but the thread scheduling there is randomness. If Thread1 is finally dispatched, then only thread1 this time Futex is called, so Pthread_join () causes the Futex call to be between 1-4 times. (although not necessarily, but 4 times more common)
So this program will at most cause 3+4+4=11 Futex system calls, with strace tracking, to verify our ideas.
19710 Futex (0xb7df1be8, futex_wait, 19711, NULL <unfinished ...>
19712 Futex (0x8049910, futex_wait, 0, NULL <unfinished ...>
19713 Futex (0x8049910, futex_wait, 0, NULL <unfinished ...>
19714 Futex (0x8049910, futex_wait, 0, NULL <unfinished ...>
19711 Futex (0x8049910, Futex_wake, 1 <unfinished ...>
19710 Futex (0xb75f0be8, futex_wait, 19712, NULL <unfinished ...>
19712 Futex (0x8049910, Futex_wake, 1 <unfinished ...>
19710 Futex (0xb6defbe8, futex_wait, 19713, NULL <unfinished ...>
19713 Futex (0x8049910, Futex_wake, 1 <unfinished ...>
19710 Futex (0xb65eebe8, futex_wait, 19714, NULL <unfinished ...>
19714 Futex (0x8049910, futex_wake, 1) = 0
(19710 is the main thread, 19711,19712,19713,19714 is 4 sub-threads)

4. More Questions
Is this the end of the story? If we try to replace semaphore with a mutex. You will find that when there is no competition from the beginning, the mutex will fully conform to the Futex mechanism, and neither lock nor unlock will invoke the Futex system call. When there is competition, the first pthread_mutex_lock will not call the Futex call, it looks normal. But the last time pthread_mutex_unlock, although no thread is waiting for the mutex, the Futex (Futex_wake) will still be called. What is the reason? Welcome to discuss!!!

Summary:
1. Although the Semaphore,mutex synchronization mode is built on the Futex synchronization mechanism. However, the limitations of semantics and so on are not fully implemented according to the original design of Futex.
2. Functions such as Pthread_join () are also called Futex to implement.
3. Different synchronization methods have their different semantics, different performance characteristics, suitable for different scenarios. We need to know their commonality in the process of use and to understand the differences between them. In order to better understand the multi-threaded scenario, write more high-quality multithreaded programs.

Reprint Address:
Http://blog.csdn.net/Javadino/archive/2008/09/06/2891385.aspx
Http://blog.csdn.net/Javadino/archive/2008/09/06/2891388.aspx

Http://blog.csdn.net/Javadino/archive/2008/09/06/2891399.aspx

Thread synchronization mechanism in Linux (iv)--C language implementation

The logic of Futex can be expressed in the following C language

int val = 0;
void Lock ()
{
int C
if ((c = Cmpxchg (val, 0, 1))! = 0) {
if (c! = 2)
c = Xchg (Val, 2);
while (c! = 0) {
Futex_wait (&val, 2);
c = Xchg (Val, 2);
}
}
}

void Unlock ()
{
if (Atomic_dec (val)! = 1)
Futex_wake (&val, 1);
}

Val 0:unlock

Val 1:lock, no waiters

Val2:lock, one or more waiters

See also: Futex is tricky

Design and implementation of "turn" Linux Futex