Implement map-reduce in the program based on the swoole task function

The swoole-based task function enables the map-reduce Swoole extension in the program. The Task process function is very powerful and can be used to implement various complex business logic. This article describes how to use the task/finish function to process the Map-Reduce concurrent tasks in the program. A chat service often has group chat requirements. In addition, members in my group and group need to be sorted by points. functions similar to this can be simply implemented using Swoole.

Traditional multi-thread solution

Create two global variables Map. group_map uses group_id as the Key and stores the member set. User_map uses uid as the Key to store all groups that the current user joins.

In a multi-threaded environment, the two maps cannot be operated directly and must be locked. When a user is added to a group or the user exits from a group, the two maps must be operated and locked. If the operation is frequent, in fact the lock collision is very serious, this part of the operation will become serial. At the same time, only one thread can operate the map. The competition for locks will also lead to a large number of thread switching, which wastes a lot of CPU resources.

lock.lock();group_map[group_id].append([uid, score]);user_map[uid].append(group_id);group_map.sortByScore();lock.unlock();

Swoole-based Task function

Based on the Swoole Task function, you can slice the Task and ship it to different Task processes by hash to complete the Task. The sorting function can be directly used by PHPSplHeapImplementation: The time complexity is O (logn). to implement the query function, for example, query all groups that the user joins based on the UID and query the members according to the GroupId. You can calculate the Hash to find the corresponding Task process, and then send the command through task/taskwait to directly read the variable of the process to find the information.

$serv->set(array("task_worker_num" => 24));$serv->task(array("cmd" => "user", "uid" => $uid, "gid" => $gid, "score" => $score), $gid % $task_worker_num);$serv->task(array("cmd" => "group", "uid" => $uid, "gid" => $gid), $uid % $task_worker_num);class MyMaxHeap extends SplHeap{    public function compare($value1, $value2)    {        return ($value1['score'] - $value2['score']);    }}function onTask($serv, $taskId, $srcWorkerId, $data) {    static $userMap = array();    static $groupMap = array();        if ($data['cmd'] == 'group')    {        if (!isset($groupMap[$data['gid']]))        {            $groupMap[$data['gid']] = new MyMaxHeap();        }        $heap = $groupMap[$data['gid']];        $heap->insert(array("uid" => $data['uid'], "score" => $data['score']));    }    elseif ($data['cmd'] == 'user')    {        $userMap[$data['uid']][] = $data['gid'];    }}

The Task process only has array operations, so it is not blocked. you only need to enable the same number of processes as the number of CPU cores. There is no lock between processes to compete, and the performance is very good. The Swoole Task process uses UnixSocket for communication. it is a full-memory communication mode provided by the kernel without any IO, and a single read/write process can reach 1 million/second. Although the speed of directly reading variables is not fast, the performance is sufficient.

----- Great split line -----

PHP phpfamily is established by a group of reliable people and is willing to bring some spiritual food worth tasting to PHPer!

This article is released by rango exclusively authorized php rice grains. for details, please indicate the source information and the following two-dimensional code (long-press the identifiable two-dimensional code to follow ):