分布式背景工作隊列類比(Golang)

這是一個建立於 的文章，其中的資訊可能已經有所發展或是發生改變。最近研究了下 gowoker，這東西代碼少而精，Golang真是很適合實現這類東西。
我去掉參數配置,JSON,Redis這些東西，用goworker的方式做了個最簡單的實現。

  實現如下功能:
     1. worker向JobServer註冊可執行檔功能
     2. JobServer輪詢,有job就執行,沒有則繼續輪詢
     3. client向JobServer提出工作要求,並傳入參數
     4. JobServer依請求丟給worker執行(可並發或串列執行)
     5. JobServer繼續輪詢

        我弄的這個代碼很少，其中隊列用數組代替,同時省掉了很多東西,
但保留了其goroutine與channel最基礎的實現。

如果想看goworker的,可以參考下我這個,應當可以更快的弄明白goworker。

    示範例子及運結果:

//分布式背景工作隊列類比(一)//author: Xiong Chuan Liang//date: 2015-3-24package mainimport ("fmt""runtime"//"strconv""time""jobserver")func main() {runtime.GOMAXPROCS(runtime.NumCPU())fmt.Println("分布式背景工作隊列類比(一)...")//Job Serverjs := jobserver.NewJobServer()//類比Worker端註冊js.RegisterWorkerClass("mail", mailWorker)js.RegisterWorkerClass("log", sendLogWorker)js.RegisterWorkerClass("exception", paincWorker)//類比用戶端發送請求go func() {time.Sleep(time.Second * 2)js.Enqueue("mail", "xcl_168@aliyun.com", "sub", "body")js.Enqueue("test_notfound", "aaaaaaaaaaaaaaaaaaa")js.Enqueue("log", "x.log", "c.log", "l.log")//測試jobserver.PARALLEL/ORDER//for j := 0; j < 100; j++ {//js.Enqueue("mail", strconv.Itoa(j))//}time.Sleep(time.Second)js.Enqueue("exception", "try{}exception{}")time.Sleep(time.Second * 5)js.Enqueue("mail", "xcl_168@aliyun.com2", "sub2", "body2")}()//啟動服務，開始輪詢// StartServer(輪詢間隔,執行方式(並發/順序))js.StartServer(time.Second*3, jobserver.ORDER) //PARALLEL}func mailWorker(queue string, args ...interface{}) error {fmt.Println("......mail() begin......")for _, arg := range args {fmt.Println("   args:", arg)}fmt.Println("......mail() end......")return nil}func sendLogWorker(queue string, args ...interface{}) error {fmt.Println("......sendLog() begin......")for _, arg := range args {fmt.Println("   args:", arg)}fmt.Println("......sendLog() end......")return nil}func paincWorker(queue string, args ...interface{}) error {fmt.Println("......painc() begin......")panic("\n    test exception........................ \n")fmt.Println("......painc() end......")return nil}/*運行結果:分布式背景工作隊列類比(一)...[JobServer] [poll] polling......mail() begin......   args: xcl_168@aliyun.com   args: sub   args: body......mail() end......[JobServer] [poll]  test_notfound  not found......sendLog() begin......   args: x.log   args: c.log   args: l.log......sendLog() end......[JobServer] [poll] polling......painc() begin......[JobServer] [run] Panicking    test exception........................[JobServer] [poll] polling[JobServer] [poll] polling......mail() begin......   args: xcl_168@aliyun.com2   args: sub2   args: body2......mail() end......[JobServer] [poll] polling[JobServer] [poll] polling[JobServer] [poll] polling[JobServer] [poll] quit*/

     上面是順序執行的運行結果, 如果要測試並發，可以將上面代碼注釋部分開啟，JobServer執行方式更改為jobserver.PARALLEL，再執行即可。

具體的實現在下面:

 下面兩個是基本的一些定義:

package jobservertype workerFunc func(string, ...interface{}) errortype Workers struct {workers map[string]workerFunc}

package jobservertype OrdType intconst (PARALLEL = 1 << iotaORDER)

關鍵的JobServer的實現:

//分布式背景工作隊列類比(一)//author: Xiong Chuan Liang//date: 2015-3-24package jobserverimport ("fmt""runtime""sync""time")type JobServer struct {WorkersJobQueue []*WorkerClassinterval time.Durationmt       sync.Mutexord      OrdType}func NewJobServer() *JobServer {s := &JobServer{}s.workers = make(map[string]workerFunc, 0)return s}func (s *JobServer) RegisterWorkerClass(className string, f workerFunc) int {if _, found := s.workers[className]; found {return 1}s.workers[className] = freturn 0}type WorkerClass struct {ClassName stringArgs      []interface{}}func (s *JobServer) Enqueue(className string, args ...interface{}) bool {s.mt.Lock()w := &WorkerClass{className, args}s.JobQueue = append(s.JobQueue, w)s.mt.Unlock()return true}//pollerfunc (s *JobServer) poll(quit <-chan bool) <-chan *WorkerClass {jobs := make(chan *WorkerClass)go func() {defer close(jobs)for {switch {case s.JobQueue == nil:timeout := time.After(time.Second * 2)select {case <-quit:fmt.Println("[JobServer] [poll] quit")returncase <-timeout:fmt.Println("[JobServer] [poll] polling")}default:s.mt.Lock()j := s.JobQueue[0]if len(s.JobQueue)-1 <= 0 {s.JobQueue = nil} else {s.JobQueue = s.JobQueue[1:len(s.JobQueue)]}s.mt.Unlock()select {case jobs <- j:case <-quit:fmt.Println("[JobServer] [poll] quit")return}}}}()return jobs}//workerfunc (s *JobServer) work(id int, jobs <-chan *WorkerClass, monitor *sync.WaitGroup) {monitor.Add(1)f := func() {defer monitor.Done()for job := range jobs {if f, found := s.workers[job.ClassName]; found {s.run(f, job)} else {fmt.Println("[JobServer] [poll] ", job.ClassName, " not found")}}}switch s.ord {case ORDER:f()default:go f()}}func (s *JobServer) run(f workerFunc, w *WorkerClass) {defer func() {if r := recover(); r != nil {fmt.Printf("[JobServer] [run] Panicking %s\n", fmt.Sprint(r))}}()f(w.ClassName, w.Args...)}func (s *JobServer) StartServer(interval time.Duration, ord OrdType) {s.interval = intervals.ord = ordquit := signals()jobs := s.poll(quit)var monitor sync.WaitGroupswitch s.ord {case ORDER: //順序執行s.work(0, jobs, &monitor)default: //並發執行concurrency := runtime.NumCPU()for id := 0; id < concurrency; id++ {s.work(id, jobs, &monitor)}}monitor.Wait()}

         goworker中要複雜的多，但簡單來說最主要的就是實現上面的這些東西，我再另增加了個順序和並發的選項。

這個例子只能在本機跑，其它東西沒有。不過配合Redis，其它用戶端或其它語言則可以通過Redis來傳遞參數及實現隊列，

把它真正用起來。

 MAIL: xcl_168@aliyun.com

BLOG: http://blog.csdn.net/xcl168