Skynet Source Analysis 3: Message scheduling

Message dispatch is divided into two levels in the framework, a C-layer distribution, and a LUA layer distribution. This article describes the C-tier, from two aspects:

1. Control of worker threads
2. Dispatch of Mailbox

Scheduling-related code implementation in/skynet-src/skynet_mq.c,/skynet-src/skynet_start.c,/skynet-src/skynet_server.c three files, the overall is a M:N scheduler.

Control of worker threads

After the framework runs, a fixed thread is started to schedule SC (Skynet_context) in turn, and the number of threads is defined by the thread field in the configuration file, which is 4 by default. How do you control these threads in the framework? Specifically implemented in the/SKYNET-SRC/SKYNET_START.C.

On line 208, the worker thread is started:

Static intWeight[] = {         -1, -1, -1, -1,0,0,0,0,        1,1,1,1,1,1,1,1,         2,2,2,2,2,2,2,2,         3,3,3,3,3,3,3,3, }; structWorker_parm Wp[thread];  for(i=0; i<thread;i++) {wp[i].m=m; Wp[i].id=i; if(I <sizeof(weight)/sizeof(weight[0]) {wp[i].weight=Weight[i]; } Else{wp[i].weight=0; } create_thread (&pid[i+3], Thread_worker, &Wp[i]); }

Just look at the thread function Thread_worker put, in line 152:

1 Static void*2Thread_worker (void*p) {3     structWorker_parm *WP =p;4     intid = wp->ID;5     intWeight = wp->weight;6     structMonitor *m = wp->m;7     structSkynet_monitor *SM = m->M[id];8 Skynet_initthread (thread_worker);9     structMessage_queue * q =NULL;Ten      while(!m->quit) { OneQ =skynet_context_message_dispatch (SM, q, weight); A         if(q = =NULL) { -             if(Pthread_mutex_lock (&m->mutex) = =0) { -+ + m->sleep; the                 //"Spurious wakeup" is harmless, -                 //because Skynet_context_message_dispatch () can is call at any time. -                 if(!m->quit) -Pthread_cond_wait (&m->cond, &m->mutex); +--M->sleep; -                 if(Pthread_mutex_unlock (&m->mutex)) { +fprintf (stderr,"Unlock Mutex error"); AExit1); at                 } -             } -         } -     } -     returnNULL; -}

The work thread that controls this life cycle is consistent with the process, with two details: 1. Evenly and non-repetitive assignment tasks. 2, not idling, the most hours delay. The former handles thread synchronization as well. Let's take a look at how Skynet handles the latter:

It is used as a condition variable to handle idling, with the condition variable has two advantages: 1, let out the CPU time slice. 2, from the external decision when wake up, so that can be awakened when there is a task, can not only maximize the non-idling, but also to reduce the processing task delay.

The implementation is the standard application of the conditional variable, and the example of the UNIX advanced programming condition variable is almost the same. There is also a count of sleep, what is the use of it? Used to determine if you want to call pthread_cond_signal.

Finally there is the question of where the waiting thread is awakened. Awakened in the socket thread and the timer thread, which is called once with the socket message, which wakes up once per refresh time.

Dispatch of Mailbox

In the previous article, we saw a Message_queue type field in SC, which is the mailbox. Skynet uses two kinds of queues to store messages and complete schedules, called Level 12 queues, a 1-level queue is a single-linked list, each node is a 2-level queue, and a 2-level queue (Message_queue) is an auto-extended circular queue that stores messages. These two queues are implemented in/SKYNET-SRC/SKYNET_MQ.C, which is very simple, and does not use a complex lock-free structure, but a spin lock to ensure thread-safe linked lists, circular queues.

The dispatch of the mailbox is 12-level queue scheduling, the overall structure is described as follows:

while (1) {

Level 1 team list;

Dispatch level 2 queue;

Level 1 team included in the team;

}

This is partially implemented in the/skynet-src/skynet_server 275 line Skynet_context_message_dispatch ():

1 structMessage_queue *2Skynet_context_message_dispatch (structSkynet_monitor *SM,structMessage_queue *q,intweight) {3     if(q = =NULL) {4Q =Skynet_globalmq_pop ();5         if(q==NULL)6             returnNULL;7     }8 9uint32_t handle =Skynet_mq_handle (q);Ten  One     structSkynet_context * CTX =Skynet_handle_grab (handle); A     if(CTX = =NULL) { -         structdrop_t d ={handle}; -Skynet_mq_release (q, Drop_message, &d); the         returnSkynet_globalmq_pop (); -     } -  -     inti,n=1; +     structskynet_message msg; -  +      for(i=0; i<n;i++) { A         if(Skynet_mq_pop (q,&msg)) { at skynet_context_release (CTX); -             returnSkynet_globalmq_pop (); -}Else if(i==0&& Weight >=0) { -n =skynet_mq_length (q); -N >>=weight; -         } in         intOverload =skynet_mq_overload (q); -         if(overload) { toSkynet_error (CTX,"May overload, Message Queue Length =%d", overload); +         } -  the Skynet_monitor_trigger (SM, Msg.source, handle); *  $         if(CTX-&GT;CB = =NULL) {Panax Notoginseng Skynet_free (msg.data); -}Else { theDispatch_message (CTX, &msg); +         } A  theSkynet_monitor_trigger (SM,0,0); +     } -  $ASSERT (q = = ctx->queue); $     structMessage_queue *NQ =Skynet_globalmq_pop (); -     if(NQ) { -         //If Global MQ is isn't empty, push Q back, and return next queue (NQ) the         //Else (Global mq is empty or block, don ' t-push Q back, and return Q again (for next dispatch) - Skynet_globalmq_push (q);WuyiQ =NQ; the     }  - skynet_context_release (CTX); Wu  -     returnQ; About}

The function is to dispatch an incoming level 2 queue and return to the next scheduled Level 2 queue. In the above implementation, there are four details:

1, 22-24 lines, when the 2 ranks are empty and did not press into the 1-level queue, then it disappeared? No, this is to reduce the idling 1-level queue, then when the 2-level queue is pressed back? In the Message_queue, there is a

If the IN_GLOBAL flag is in a Level 1 queue, when the 2-level queue's outbound (Skynet_mq_pop) fails, the tag is set to 0, and when the 2 team is included (Skynet_mq_push) it will be judged, and if it is 0, it will be pressed into the 1-level queue. (Skynet_mq_mark_release will also judge) so this level 2 queue will be pressed back the next time you join the team.

2, 25-27, modify the for loop number of times, that is, each time the dispatch processing how many messages. This number is related to the incoming weight, and we look back to see where this weight came from, at the source when the worker thread was created:

static int weight[] = {         -1,-1,-1,-1, 0, 0, 0, 0,        1, 1, 1, 1, 1, 1, 1, 1,         2, 2, 2, 2, 2, 2, 2, 2, 3, 3,         3, 3, 3, 3, 3, 3,};    struct Worker_parm wp[thread];    for (i=0;i<thread;i++) {        wp[i].m = m;        Wp[i].id = i;        if (I < sizeof (weight)/sizeof (Weight[0])) {            wp[i].weight= weight[i];        } else {            wp[i].weight = 0;        }        Create_thread (&pid[i+3], Thread_worker, &wp[i]);    }

Then take a look at N >>= weight, well, roughly is: the work thread into groups, the first four groups of 8 per group, more than the 5th Group, AE group each scheduling processing a message, B group each processing (N/2) bar, Group C each processing (N/4) bar, Group D each processing (N/8) bar. is for uniform use of multicore.

3, 29-32 did a load judgment, the load threshold is 1024. But it's just the output of a log reminder.

4, 34, 42 triggered a monitor, this monitoring is used to detect whether the message processing has a dead loop, but also just output a log to remind. This test is done in a dedicated monitoring thread, and the time to judge the dead loop is 5 seconds. The specific mechanism is not said here, in fact, now/SKYNET-SRC/SKYNET_MONITOR.C

Skynet Source Analysis 3: Message scheduling