A Golang to realize multi-process task processing

This is a creation in Article, where the information may have evolved or changed. # a Golang to implement multi-process task Processing # # So what kind of task is it, generally in the producer-consumer model of the consumer process, to cite a few examples 1. Consumption Kafka data 2. Consume Redis Data 3. Poll processing database data 4. ... # # # below to analyze 1. How many business logic processing processes are there? Processing a data on go one, also can but a little rough, the process is not the more the better, scheduling is also good performance, so or control, general bar to get a CPU * 2 is almost the same (runtime. NUMCPU () 2. Get Data Association Because the example I want to analyze is a for loop that reads data over and over to a task-handling coprocessor, so here's a co-process 3. Process how to shut down can not kill-9 rough treatment, so easy to cause data anomalies or data loss, is generally captured signal signal. Notify (Signalchan, Syscall. SIGINT, Syscall. SIGTERM) directly on the code ' Golangpackage mainimport ("FMT" "OS" "Os/signal" "Runtime" "Sync/atomic" "Syscall" "Time") type Taskdata struct {}type Service struct {capacity inttasks chan *taskdatanumthread Intclosechans Chan Struct{}stopflag int32loopstop Chan Chan struct{}}func newservice (capacity int) *service {Service: = &service{}service.capacity = Capacityservice.numthread = runtime. NUMCPU () * 2service.tasks = Make (chan *taskdata, capacity) Service.stopflag = 0service.closechans = Make (chan struct{}, Ser Vice.numthread) Service.loopstopchan = Make (chan struct{}) return Service}func (This *seRvice) Stop () {Atomic. StoreInt32 (&this.stopflag, 1) <-this.loopstopchanclose (this.tasks) for I: = 0; i < This.numthread; i++ {<-this.closechans}}func (this *service) Run () {for I: = 0, i < This.numthread; i++ {go This.run (i)}go this. Loopconsume ()}func (this *service) run (i int) {FMT. Println ("Go run:", i) loop:for {select {case task, OK: = <-this.tasks:if OK {//#TODO processfmt. PRINTLN ("Process", Task)} else {break Loop}}}this.closechans <-Struct{}{}}func (this *service) Loopconsume () {fmt. Println ("Loop") for atomic. LoadInt32 (&this.stopflag) = = 0 {//todo Readdatatask: = &taskdata{}this.tasks <-taskfmt. Println ("consume.") Time. Sleep (time. Second * 2)}this.loopstopchan <-Struct{}{}}func Main () {service: = NewService (+) Go service. Run ()//Startup program handles C: = Make (chan os. Signal) Signal. Notify (c, OS. Interrupt, Syscall. SIGINT, Syscall. SIGTERM) S: = <-c//waits to close the signal fmt. PRINTLN (s) service. Stop ()//Turn off service FMT. Println ("Exit:D")} ' # # 1: ' ' Golangservice.stopflag serviCe.closeChansservice.loopStopChan ' ' These variables are used for safe Exit program with # # 1. Stopflag> first to exit Loopconsume cycle with what to notice, with channel also can, is to cooperate with select Use, but with atomic mark is not more concise? So Stopflag is to quit the Loopconsume with # # # 2. Closechans> because we go a lot of the process, then to listen to each of the process exit, you need multiple channel to receive ' for I: = 0; i < This.numthread; i++ {<-this.closechans} ' > This code means waiting for all processing to exit # # # 3. Loopstopchan> This is what to do, but also to deal with the exit of the association is just loopconsume, why not Closechans size and then add one and become this "Service.closechans = Make (chan struct{}, Service.numthread+1) "" Func (This *service) Stop () {Atomic. StoreInt32 (&this.stopflag, 1) Close (This.closechans) for I: = 0; i < this.numthread+1; i++ {<-this.closechans}} "What will happen?" , if this happens once stop is This.stopflag = 1, but Loopconsume may still be in the data phase from//todo ReadData when Close (this.tasks) is executed, and then it is time to execute This.task S <-task, but at this time the tasks have been closed, then will panic in fact in the entire example loopconsume is equivalent to a producer, and run is the equivalent of a consumer, we should not first shut the producers wait for the consumer to quit after consumption, no doubt certainly Yes, so there's going to be a channel, and so the producer exits and sends the channel to let the consumer out, so alone with a loopstopchan## think 2: "FuNC (This *service) Loopconsume () {fmt. Println ("Loop") for atomic. LoadInt32 (&this.stopflag) = = 0 {//todo Readdatatask: = &taskdata{}this.tasks <-taskfmt. Println ("consume.") Time. Sleep (time. Second * 2)}this.loopstopchan <-struct{}{}} "This code is actually to get the data, I do not write the part of the data, because this is business-related, for example, such as a real point Read the Mysqlselect ID, * from data WHERE ID > Offset LIMIT n, read N data from offset position each time, if the number of bars obtained after reading is NUM, and if Num equals N, then OFFSET + = N continues read, otherwise the data is insufficient, then OFFSET + = num, and sleep N seconds (to avoid the absence of data when the empty run) on the pseudo Code "func (This *service) Loopconsume () {fmt. Println ("Loop") for atomic. LoadInt32 (&this.stopflag) = = 0 {rows:= Read (offset) if rows rows = = N {task: = &taskdata{}this.tasks <-Taskoffset + = N}else{time. Sleep (time. Second *) Offset + = rows rows}}this.loopstopchan <-struct{}{}} "" There is no problem here, if the program just entered time. Sleep (time. Second * 20) Here, at this time stop is not to wait 20s but in fact the process has been very idle, there is no way to solve it, or can be marked with a method, a program into sleep can be set a mark on the pseudo code "" Func (this *service) loopc Onsume () {fmt. Println ("Loop") for atomic. LoadInt32 (&thiS.stopflag) = = 0 {atomic. StoreInt32 (&this.forcestopflag, 0) rows:= Read (offset) If rows row number = = N {task: = &taskdata{}this.tasks <-Taskoff Set + = N}else{atomic. StoreInt32 (&this.forcestopflag, 1) time. Sleep (time. Second *) Offset + = rows rows}}this.loopstopchan <-struct{}{}}func (this *service) Stop () {Atomic. StoreInt32 (&this.stopflag, 1) if atomic. LoadInt32 (&this.forcestopflag) = = 0{<-this.loopstopchan//only when Forcestopflag = 0 is required to wait for Loopconsume to exit}close ( This.tasks) for I: = 0; i < This.numthread; i++ {<-this.closechans}} "' # # # 3: This model run inside or loopconsume can also go out, obviously not, because once go out, then the existing stop is invalid, because can not get these processes are Exit. In fact, I think there is no need to go out again, because Loopconsume is generally read, faster than the business processing of higher, if this does not meet your actual business needs, you can go to multiple loopconsume, the same loopstopchan also into the length of the CH Annel<-this.loopstopchan becomes this "for I: = 0; I <n; i++ {<-this.loopstopchan} "' Again extreme your business has to be in the loopconsume go one or more, then you have to think about how to synchronize, as to what method, have to think about, you can propose to discuss the same with the same in the run Inside you also want to go one or more co-process, or the same way to think of a synchronization problem # # Summarize this small service onlyis a multi-process processing task routines, commonly used in the consumer process of producer consumer models. For high performance requirements may not be suitable, such as the number of threads here is fixed, you can improve the dynamic changes in the expansion, code is relatively simple to write, some errors also hope that the great God many correct, welcome to discuss. 766 reads  ∙  1 likes