Efficient development experience in multi-thread programming on Linux

April 23, 2009

In this article, we summarize five experiences based on the main features of multi-threaded programming in Linux to improve the habit of multi-threaded programming in Linux and avoid development traps. In this article, we interspersed with some windows programming cases to compare Linux features, to enhance the reader's impression.

Background

The multi-threaded program development on the Linux platform is slightly different from the multi-threaded APIs on other platforms (such as Windows. Not paying attention to some development traps on these Linux systems, it often causes program problems and deadlocks. In this article, we will summarize the LINUX multi-threaded programming problems from five aspects, and draw out relevant development experiences to avoid these traps. We hope that these experiences will help readers better and more quickly get familiar with multi-threaded programming on the Linux platform.

We assume that all readers are familiar with the pthread library API of basic thread programming on the Linux platform. Other third-party libraries used for thread programming, such as boost, will not be mentioned in this article. This article covers thread management, mutex variables, and condition variables in thread development. The process concept will not be covered in this article.

 

Back to Top

Introduction to thread-based API development on Linux

Multi-thread development has mature pthread library support on the Linux platform. The most basic concepts involved in multi-threaded development include three main points: threads, mutex locks, and conditions. Among them, thread operations are divided into three types: thread creation, exit, and wait. Mutex locks include creation, destruction, locking, and unlocking. Conditional operations include creation, destruction, triggering, broadcast, and waiting. Other thread extension concepts, such as signal lights, can be encapsulated through the basic operations of the above three basic elements.

Threads and mutex locks. The corresponding API of the condition on the Linux platform can be summarized in table 1. To make it easier for users familiar with Windows Thread Programming to get familiar with LINUX multi-thread development APIs, we also list the corresponding API names in the Windows SDK library in the table.

Table 1. thread function list

Object	Operation	Linux pthread API	Windows SDK library APIs
Thread	Create	Pthread_create	Createthread
	Exit	Pthread_exit	Threadexit
	Wait	Pthread_join	Waitforsingleobject
Mutex lock	Create	Pthread_mutex_init	Createmutex
	Destroy	Pthread_mutex_destroy	Closehandle
	Lock	Pthread_mutex_lock	Waitforsingleobject
	Unlock	Pthread_mutex_unlock	Releasemutex
Condition	Create	Pthread_cond_init	Createevent
	Destroy	Pthread_cond_destroy	Closehandle
	Trigger	Pthread_cond_signal	Setevent
	Broadcast	Pthread_cond_broadcast	Setevent/resetevent
	Wait	Pthread_cond_wait/pthread_cond_timedwait	Singleobjectandwait

 

Multi-thread development has mature pthread library support on the Linux platform. The most basic concepts involved in multi-threaded development include three main points: threads, mutex locks, and conditions. Among them, thread operations are divided into three types: thread creation, exit, and wait. Mutex locks include creation, destruction, locking, and unlocking. Conditional operations include creation, destruction, triggering, broadcast, and waiting. Other thread extension concepts, such as signal lights, can be encapsulated through the basic operations of the above three basic elements.

 

Back to Top

Five experiences in Linux Thread Programming

Set the recursive attribute whenever possible to initialize Linux mutex Variables

Mutex lock is a basic concept in multi-threaded programming and is widely used in development. The Calling sequence is clear and simple: Lock creation, lock adding, unlock, and lock destruction. However, unlike mutex variables on Windows, the same thread in Linux cannot recursively accelerate the same mutex lock by default. Otherwise, a deadlock may occur.

Recursive locking means attempts to perform two or more actions on the mutex lock in the same thread. The code for the scenario in Linux can be shown in Listing 1.

Listing 1. Repeated instance lock for mutex lock in Linux

// Create the lock pthread_mutex_t * themutex = new pthread_mutex_t; pthread_mutexattr_t ATTR; trim (& ATTR); pthread_mutex_init (themutex, & ATTR); trim (& ATTR ); // recursively lock pthread_mutex_lock (themutex); pthread_mutex_unlock (themutex );

 

In the above Code scenario, the problem occurs in the second lock operation. By default, Linux does not allow recursive locking of the same thread, so the thread will experience a deadlock during the second locking operation.

The strange behavior of Linux mutex variables may be useful in some specific scenarios, but in most cases it looks more like a program bug. After all, recursive locking of the same mutex lock in the same thread is often required in secondary development.

This issue is related to the default recursive attribute in the mutex. The solution is to explicitly set the recursive attribute when the mutex variable is initialized. Based on this, the above Code can run well with slight modifications. You only need to add an attribute when initializing the lock. See list 2.

List 2. Set the recursive attribute instance of the mutex lock

Pthread_mutexattr_init (& ATTR); // set the recursive attribute pthread_mutexattr_settype (& ATTR, pthread_mutex_recursive_np); pthread_mutex_init (themutex, & ATTR );

 

Therefore, we recommend that you set the recursive attribute to initialize the Linux mutex lock. This can solve the recursive locking problem of the same thread and avoid deadlocks in many cases. Another advantage of doing so is to unify the performance of locks in Windows and Linux.

Pay attention to the automatic resetting of trigger condition variables on Linux.

There are two common models for the placement and resetting of conditional variables: the first model is that when the conditional variable is set to signaled, if no thread is waiting currently, its status remains set to signaled until a waiting thread enters the triggered state and its status changes to unsignaled ), this model is represented by the auto-set event on the Windows platform. Its status changes 1:

Figure 1. Windows conditional variable status change process

The second model is the model used by pthread on the Linux platform. When the conditional variable is set to signaled, its status will be restored to unsignaled even if no thread is waiting) status. The status change is as follows:

Figure 2. Linux conditional variable status change process

Specifically, the condition variable state change model under pthread on Linux works like this: When pthread_cond_signal () is called to release a thread that is blocked by the condition, no matter whether there is a thread that is blocked, the condition will be reset again, And the next thread blocked by the condition will not be affected. In Windows, when setevent is called to trigger the event condition of auto-reset, if the thread is not blocked by the condition, the condition will remain in the trigger state, until a new thread is blocked by conditions and released.

This difference may cause unexpected embarrassing results for programmers who are familiar with the conditional variable state model on the Windows platform and want to develop multithreading on the Linux platform. Imagine a procedure for passengers to take a taxi: passengers waiting for a taxi on the side of the road, call conditions to wait. When a taxi arrives, it will trigger conditions, passengers stop waiting and get on the bus. A taxi can carry only one wave of passengers, so we use a single trigger condition variable. This implementation logic is in the first model, even if a taxi arrives first, there will be no problem, as shown in process 3:

Figure 3. Taxi instance process using the Windows Conditional Variable Model

However, if you follow this idea in Linux, the Code may look like this in listing 3.

Listing 3. Linux taxi case code example

...... // Prompt the condition variable pthread_cond_t taxicond; // Synchronous lock pthread_mutex_t taximutex; // passengers arrive at the waiting taxi void * traveler_arrive (void * Name) {cout <"Traveler: "<(char *) name <" needs a taxi now! "<Endl; pthread_mutex_lock (& taximutex); pthread_cond_wait (& taxicond, & taxtmutex); Combine (& taxtmutex); cout <" Traveler: "<(char *) name <"now got a taxi! "<Endl; pthread_exit (void *) 0);} // The taxi arrived at Void * taxi_arrive (void * Name) {cout <" taxi "<(char *) name <"arrives. "<Endl; pthread_cond_signal (& taxtcond); pthread_exit (void *) 0);} void main () {// initialize taxtcond = signature; taxtmutex = signature; pthread_t thread; export threadattr; pthread_attr_init (& threadattr); pthread_create (& Thread, & threadattr, taxt_arrive, (void *) ("Jack"); sleep (1); pthread_create, & threadattr, traveler_arrive, (void *) ("Susan"); sleep (1); pthread_create (& Thread, & threadattr, taxi_arrive, (void *) ("Mike"); sleep (1); Return 0 ;}

 

Okay. Run the command and check the result in Listing 4.

Listing 4. Program result output

Taxi Jack arrives. Traveler Susan needs a taxi now! Taxi Mike arrives. Traveler Susan now got a taxi.

 

The process 4 is shown below:

Figure 4. Taxi instance process using the Linux Conditional Variable Model

By comparing the results, you will find that the results of running the same logic on the Linux platform are completely different. For Model 1 on Windows, Jack drove a taxi to the platform to trigger the condition variable. If there are no customers, the condition variable will remain in the trigger state, that is, Jack stops waiting there. Until Miss Susan came to the platform and waited for a taxi. Susan took Jack's taxi and the condition variable was automatically reset.

But on the Linux platform, the problem came. Jack went to the platform and saw no one. The trigger condition variable was directly reset, so Jack waited in the queue. Miss Susan arrived at the platform a second later, but she could not see Jack waiting there. She had to wait until Mike arrived and re-triggered the conditional variable. Susan got into Mike's car. This is unfair to Jack in front of the queuing system, and the crux of the problem is a bug caused by automatic reset triggered by conditional variables on the Linux platform.

This model of conditional variables on Linux is hard to say good or bad. But in actual development, We Can slightly improve the Code to avoid this difference. Because this difference only occurs when the trigger is not waiting by the thread at the time of the condition variable, we only need to grasp the timing of the trigger. The simplest way is to add a counter to record the number of waiting threads and check the variable before determining the trigger condition variable. The improved Linux functions are shown in listing 5.

Listing 5. Linux taxi case code example

...... // The variable pthread_cond_t taxicond indicating the arrival of a taxi; // the synchronized lock pthread_mutex_t taximutex; // Number of passengers, initially 0 int travelercount = 0; // passengers arrive at the waiting taxi void * traveler_arrive (void * Name) {cout <"Traveler:" <(char *) name <"needs a taxi now! "<Endl; pthread_mutex_lock (& taximutex); // The number of passengers is increased by travelercount ++; pthread_cond_wait (& taxicond, & taximutex); Combine (& taximutex ); cout <"Traveler:" <(char *) name <"now got a taxi! "<Endl; pthread_exit (void *) 0);} // The taxi arrived at Void * taxi_arrive (void * Name) {cout <" taxi "<(char *) name <"arrives. "<Endl; while (true) {pthread_mutex_lock (& taximutex); // The condition variable if (travelercount> 0) is triggered only when a passenger is found waiting) {pthread_cond_signal (& taxtcond); pthread_mutex_unlock (& taximutex); break;} pthread_mutex_unlock (& taximutex);} pthread_exit );}

 

Therefore, we recommend that you check whether there is a waiting thread before you start the conditional variable on the Linux platform. The conditional variable is triggered only when there is a thread waiting.

Note the issue of unlocking mutex lock when the condition is returned.

When Linux calls pthread_cond_wait to wait for the conditional variable operation, it is necessary to add a mutex variable parameter to avoid competition and hunger between threads. However, when the condition is waiting for the return result, you must note that the mutex variable must not be unlocked.

When the pthread_cond_wait (pthread_cond_t * cond, pthread_mutex_t * mutex) function returns, the mutex lock is locked. Therefore, if you need to re-access the data in the critical section, there is no need to re-lock the mutex. However, the following problem arises: after each conditional wait, you need to add a manual unlock operation. As shown in Listing 6, the Linux code for passengers waiting for a taxi in the previous article is as follows:

Listing 6. The unlocked instance after the condition variable is returned

void * traveler_arrive(void * name) { cout<< ” Traveler: ” <<(char *)name<< ” needs a taxi now! ” <<endl; pthread_mutex_lock(&taxiMutex); pthread_cond_wait (&taxiCond, &taxtMutex); pthread_mutex_unlock (&taxtMutex); cout<< ” Traveler: ” << (char *)name << ” now got a taxi! ” <<endl; pthread_exit( (void *)0 ); }

 

This is especially important for developers familiar with Windows platform multithreading development. The signalobjectandwait () function on Windows is usually equivalent to the pthread_cond_wait () function on Linux. However, you must note that the two functions exit in different States. On Windows, signalobjectandwait (handle a, handle B ,...... ) When a method is returned at the end of a call, the States A and B are both signaled. in common usage, A is often a mutex variable. In this case, when returned, mutex A is in the signaled status and Event B is in the signaled status. Therefore, for mutex A, we do not need to consider unlocking. In addition, after signalobjectandwait (), if you need to re-access the data in the critical section, you must call waitforsingleobject () to re-lock the lock. This is exactly the opposite of pthread_cond_wait () in Linux.

Linux is very important for this extra unlock operation on Windows. Otherwise, the conditional wait operation transplanted from windows to Linux will surely experience a deadlock once the unlock operation is forgotten.

Absolute Waiting Time

Timeout is a common concept in multi-threaded programming. For example, when you use pthread_cond_timedwait () on Linux, You need to specify the timeout parameter so that the caller of this API can only be blocked for a specified interval. However, if this is the first time you use this API, the first thing you need to know is the particularity of the timeout parameter in this API (as prompted by the title of this section ). Let's first take a look at the definition of this API. For the pthread_cond_timedwait () definition, see listing 7.

Listing 7. pthread_cond_timedwait () function definition

int pthread_cond_timedwait(pthread_cond_t *restrict cond, pthread_mutex_t *restrict mutex, const struct timespec *restrict abstime);

 

The parameter abstime is used to represent a parameter related to the timeout time, but it must be noted that it represents an absolute time, rather than a time interval value, a timeout event is triggered only when the current time of the system reaches or exceeds the time indicated by abstime. This may be especially confusing for people with experience in thread development on the Windows platform. Because all the API wait parameters (such as signalobjectandwait) in windows are relative time,

Assume that we specify a relative timeout parameter such as dwmilliseconds (unit: milliseconds) to call timeout-related functions. In this case, we need to convert dwmilliseconds to the absolute time parameter abstime in Linux. Common conversion methods are shown in listing 8:

Listing 8. Converting instances from relative time to absolute time

/* get the current time */ struct timeval now; gettimeofday(&now, NULL); /* add the offset to get timeout value */ abstime ->tv_nsec = now.tv_usec * 1000 + (dwMilliseconds % 1000) * 1000000; abstime ->tv_sec = now.tv_sec + dwMilliseconds / 1000;

 

Linux's absolute time seems simple and clear, but it is a very obscure trap in development. And once you forget the time conversion, you can imagine how troublesome it will be to wait for your mistake: if you forget to convert relative time to absolute time, it is equivalent to telling the system that the wait timeout period is a time period in the past month, January 1, 1970, so the operating system does not hesitate to give you a timeout return value immediately, then you will raise your fist to complain about why another synchronization thread took so long and plunged into the abyss to find the cause of time consumption.

Correctly handle thread termination issues in Linux

On the Linux platform, when the processing thread ends, you need to pay attention to the problem of how to make a thread start and end, so that the occupied resources are correctly released. In Linux, although each thread is independent of each other, termination of one thread will not be notified or affect other threads. However, the resources of a terminated thread will not be released with the termination of the thread. We need to call pthread_join () to obtain the termination status of another thread and release the resources occupied by the thread. The pthread_join () function is defined in listing 9.

Listing 9. pthread_join Function Definition

int pthread_join(pthread_t th, void **thread_return);

 

The thread that calls this function will be suspended, waiting for the end of the thread indicated by Th. Thread_return is a pointer to the return value of the thread th. Note that the thread indicated by th must be joinable, that is, it is in a non-detached (free) state, and only one thread can call pthread_join () on th (). If TH is in the detached state, an error is returned for the call to th's pthread_join.

If you don't care about the end state of a thread, you can also set a thread to the detached state to allow the operating system to recycle its resources at the end of the thread. You can set a thread to the detached status in two ways. One is to call the pthread_detach () function and set the thread th to the detached state. Its statement is shown in listing 10.

Listing 10. pthread_detach Function Definition

int pthread_detach(pthread_t th);

 

Another method is to set the thread to the detached state when it is created. First, initialize a thread attribute variable and set it to the detached state, finally, pass it as a parameter to the thread creation function pthread_create (), so that the created thread is directly in the detached state. The method is shown in listing 11.

Listing 11. Creating a detach thread code instance

………………………………… .. pthread_t tid; pthread_attr_t attr; pthread_attr_init(&attr); pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED); pthread_create(&tid, &attr, THREAD_FUNCTION, arg);

 

In short, in order to avoid the correct release of the thread resources at the end of the thread when pthread is used, to avoid potential memory leakage, when the thread ends, to ensure that the thread is in the detached state, you must call the pthread_join () function to recycle its resources.

 

Back to Top

Summary and supplement

The above section details five efficient development experiences of multi-threaded programming in Linux. In addition, you can try other open-source class libraries for thread development.

1. Boost Library

The boost library is initiated by a member of the class library working group of the C ++ Standards Committee and is dedicated to the boost organization for C ++ to develop new class libraries. Although the library is not developed for multithreading, it has provided comprehensive API support for multithreaded programming. The boost library is more similar to the Linux pthread library for multi-threaded APIs. The difference is that it encapsulates thread, mutex lock, conditions, and other thread development concepts into C ++ classes, to facilitate development and calling. The boost library currently supports both Windows and Linux, as well as various commercial UNIX versions. If developers want to use a highly stable unified thread programming interface to reduce the difficulty of cross-platform development, boost libraries will be the first choice.

2. Ace

Ace stands for adaptive communication environment. It is a free, open-source, object-oriented tool framework for developing concurrent access software. Since Ace was originally designed for programming and development on the network server, it also provides comprehensive support for the tool library for thread development. The supported platforms are also comprehensive, including windows, Linux and various versions of UNIX. The only problem with Ace is that it seems to be too heavyweight if it is only used for Thread Programming. In addition, its complicated configurations make it easy for beginners to deploy.