Golang using Redis blocking read Brpop for instant response concurrent execution

This is a creation in Article, where the information may have evolved or changed.

Realize

The main use of Redis Brpop blocking read and Golang goroutine concurrency control and os/exec execution program, implementation of the queue with data immediately execute the corresponding program and set the results of the task key.

Operating parameters

设置brpop的超时(-t)和同步调度时返回的结果ttl(-e)./dispatchdeploy Usage:  -h 192.168.6.151 -p 6388 -t 300 -a /path/testfile.pl -e 1800  -a string        start appname (default "/path/testfile.pl")  -e int        redis expire time sec (default 1800)  -h string        redis ip  -p int        redis port (default 6379)  -t int        redis brpop timeout (default 300)

static data

const (    maxthread        = 2     //最大并发协程数    queueName        = "qn_kt"  //阻塞读队列    result_queueName = "rt_kt"  //同步返回结果的key前缀    token            = "##"    //执行调度参数的指定分隔符    sync_flag        = "1"   )

Key code

//阻塞读，当有数据分割参数，使用channel控制并发协程数，在execCmd的cmd.wait正常后释放channelfor {        content, _ := redisdb.brpop(queueName, *timeout)        if content != nil {            args := strings.Split(string(content[1]), token)            if len(args) != 4 {                log.Printf("%v lack of para length %s\n", args, len(args))            } else {                //控制并发数                sync_num <- 1                go execCmd(*appname, args, redisdb)                log.Printf("%s %v Go\n", *appname, args)            }        } else {            log.Printf("timeout %d get nil contenet , just go on", *timeout)        }    }

Test

  Lpush three dispatch to queue 127.0.0.1:6888> lpush qn_kt "6234# #ZYYC0001 # #20170620140000 # #0" "5234# #ZYYC0001 # # 20170620140000# #1 "" 7234# #ZYYC0001 # #20170620140000 # #1 "(integer) 3//control concurrency is 2, dispatch executes two Perl programs immediately, wait until the results are returned to perform a third 2017/06/20 16:45:21 Start Listen qn_kt2017/06/20 16:45:25 testfile.pl [6234 ZYYC0001 20170620140000 0] go2017/06/20 16:45:25 testfile . pl [5234 ZYYC0001 20170620140000 1] go2017/06/20 16:45:30 testfile.pl [6234 ZYYC0001 20170620140000 0] finish2017/06/20 1 6:45:30 testfile.pl [7234 ZYYC0001 20170620140000 1] go2017/06/20 16:45:30 testfile.pl [5234 ZYYC0001 20170620140000 S] fi  NISH2017/06/20 16:45:35 testfile.pl [7234 ZYYC0001 20170620140000 S] finish2017/06/20 16:45:51 timeout-get nil contenet , just go on2017/06/20 16:46:12 timeout-get nil contenet, just go on//synchronization scheduling task after the completion of the set corresponding task number, read by the interface program, 1800 seconds after the Redis recovery 127.0. 0.1:6888> get rt_kt_7234_zyyc0001 "7234# #ZYYC0001 # #20170620140000 # #1" 127.0.0.1:6888> ttl rt_kt_7234_zyyc0001 ( Integer) 1791