Implementation example of a lockless ordered linked list in C/C ++

Main problems

The main operations of the linked list include insert and remove. First, simply implement a version and you will see the problem. The following code is only used as an example:

Struct node_t {key_t key; value_t val; node_t * next;}; int l_find (node_t ** pred_ptr, node_t ** item_ptr, node_t * head, key_t key) {node_t * pred = head; node_t * item = head-> next; while (item) {int d = KEY_CMP (item-> key, key ); if (d> = 0) {* pred_ptr = pred; * item_ptr = item; return d = 0? TRUE: FALSE;} pred = item; item = item-> next;} * pred_ptr = pred; * item_ptr = NULL; return FALSE;} int l_insert (node_t * head, key_t key, value_t val) {node_t * pred, * item, * new_item; while (TRUE) {if (l_find (& pred, & item, head, key )) {return FALSE;} new_item = (node_t *) malloc (sizeof (node_t); new_item-> key = key; new_item-> val = val; new_item-> next = item; //. if pred itself is removed if (CAS (& pred- > Next, item, new_item) {return TRUE;} free (new_item) ;}int l_remove (node_t * head, key_t key) {node_t * pred, * item; while (TRUE) {if (! Rochelle find (& pred, & item, head, key) {return TRUE;} // B. if pred is removed, if item is also removed, if (CAS (& pred-> next, item, item-> next) {haz_free (item); return TRUE ;}}}

The Rochelle find function returns the pre-ordered elements and elements themselves. Although Code A and B get pred and item, it may be removed by other threads during CAS. Even every element of Rochelle find may be removed. The problem is that when an element is obtained at any time, it is not sure whether it is still valid. The validity of an element includes whether it is still in the linked list and whether the memory it points to is still valid.

Solution

By adding a valid flag for the element pointer and cooperating with the mutual exclusion of CAS operations, the problem of determining the element validity can be solved.

Because node_t is aligned in the memory, the pointer to node_t is not used for a few lower bits, so that the flag can be set in the lower bits, in this way, the effect of DCAS is achieved during CAS, which is equivalent to converting two logical operations into one atomic operation. Imagine the thread security of the reference counting object. The pointer encapsulated in it is thread-safe, but the object itself is not.

CAS is mutually exclusive. When CAS has the same object in several threads, only one thread succeeds, and the failed thread can determine that the target object has changed. Improved code (the code is only used as an example and is not guaranteed to be correct ):

Typedef size_t markable_t; // The lowest position 1, indicating that the element is deleted # define HAS_MARK (p) (markable_t) p & 0x01) # define MARK (p) (markable_t) p | 0x01) # define STRIP_MARK (p) (markable_t) p &~ 0x01) int l_insert (node_t * head, key_t key, value_t val) {node_t * pred, * item, * new_item; while (TRUE) {if (l_find (& pred, & item, head, key) {return FALSE;} new_item = (node_t *) malloc (sizeof (node_t); new_item-> key = key; new_item-> val = val; new_item-> next = item; //. although find obtains a valid pred, the pred may be deleted before the following code. In this case, pred-> next is marked // pred-> next! = Item. The CAS fails. if (& pred-> next, item, new_item) {return TRUE;} free (new_item );} return FALSE;} int l_remove (node_t * head, key_t key) {node_t * pred, * item; while (TRUE) {if (! Rochelle find (& pred, & item, head, key) {return FALSE;} node_t * inext = item-> next; // B. mark item-> next before deleting an item. if CAS fails, another thread deletes the item after find. if so, retry if (! CAS (& item-> next, inext, MARK (inext) {continue;} // C. when deleting an item of the same element, only one thread will successfully go here if (CAS (& pred-> next, item, STRIP_MARK (item-> next ))) {haz_defer_free (item); return TRUE ;}} return FALSE ;}int l_find (node_t ** pred_ptr, node_t ** item_ptr, node_t * head, key_t key) {node_t * pred = head; node_t * item = head-> next; hazard_t * hp1 = haz_get (0); hazard_t * hp2 = haz_get (1); while (item) {haz_set_ptr (Hp1, pred); haz_set_ptr (hp2, item);/* if it has been marked, the item may be removed or even released from the linked list, so you need to resend the search */if (HAS_MARK (item-> next) {return l_find (pred_ptr, item_ptr, head, key);} int d = KEY_CMP (item-> key, key); if (d> = 0) {* pred_ptr = pred; * item_ptr = item; return d = 0? TRUE: FALSE;} pred = item; item = item-> next;} * pred_ptr = pred; * item_ptr = NULL; return FALSE ;}

Functions such as haz_get and haz_set_ptr are implemented by hazard pointer to support GC in multi-thread memory. In the code above, when you want to delete an element item, it will mark item-> next, so that the CAS in insert does not need to be adjusted. To sum up the thread competition:

In insert, find the normal pred and item, pred-> next = item, and then a thread before CAS deletes the pred. In this case, pred-> next = MARK (item ), CAS fails and retries. Deletion can be divided into two types: a) remove from the linked list, get the mark, and the pred can continue to be accessed; B) the pred may be released, and then use the pred will be wrong. To handle situation B, a mechanism similar to hazard pointer is introduced to effectively ensure that the memory of any pointer p will not be actually released as long as there are threads using it.
Multiple threads insert elements after the pred in the insert statement, which is also guaranteed by the CAS in the insert statement.
In remove, the conditions are the same as insert. find obtains valid pred and next, but in CAS, pred is deleted by other threads. In this case, insert and CAS fail. Retry.
When you change the structure of a linked list, you need to retry this operation, whether it is remove or insert.
In "find", there may be items marked to be deleted. In this case, items may be deleted according to the remove implementation, so you need to traverse them again.
ABA problems

The ABA problem still exists. In insert:

If (CAS (& pred-> next, item, new_item) {return TRUE ;}

If the item after pred is removed before CAS and added with the same address value, but its value changes, CAS will succeed, but the linked list may not be ordered. Pred-> val <new_item-> val> item-> val

To solve this problem, you can store a count by using the other bits of the pointer value address alignment to indicate the number of changes to pred-> next. When the insert operation obtains the pred, the count stored in pred-> next is assumed to be 0. Before CAS, other threads removed the pred-> next and added the item, in this case, the count in pred-> next is increased, leading to CAS failure in insert.

// The delimiter is reserved as the delete flag # define MASK (sizeof (node_t)-1 )&~ 0x01) # define GET_TAG (p) (markable_t) p & MASK) # define TAG (p, tag) (markable_t) p | (tag )) # define MARK (p) (markable_t) p | 0x01) # define HAS_MARK (p) (markable_t) p & 0x01) # define STRIP_MARK (p) (node_t *) (markable_t) p &~ (MASK | 0x01) implementation of remove:/* mark and then delete */if (! CAS (& sitem-> next, inext, MARK (inext) {continue;} int tag = GET_TAG (pred-> next) + 1; if (CAS (& pred-> next, item, TAG (STRIP_MARK (sitem-> next), tag) {haz_defer_free (sitem); return TRUE ;}

You can also update the count of pred-> next in insert.

Summary

The implementation of lock-free is essentially dependent on the mutual exclusion of CAS. Implement a data structure of lock free from the beginning, and you can deeply feel the tricky implemented by lock free. The final code can be obtained from github. In the code, a not very powerful hazard pointer is implemented for simplicity.