Sidekiq in Rails

Sidekiq in Rails
Introduction

Sidekiq is a multi-threaded background task processing system. Before it came out, a Resque product (made on Github) exists, because Resque is a multi-process model, therefore, when the number of tasks increases, the memory resources consumed by the Resque will be very large, so we can use Celluoid to complete the Reqsue in the form of multithreading, that is, today's Sidekiq (2.15.xx)

Basic Architecture Job

In Sidekiq, a Job refers to the execution of a task. Note that in our general sense, "a Job" refers to a type of Job.

Worker

BecauseSidekiqCelluoid is used to implement multi-thread control, while Celluoid is implemented in the multi-thread mode Actor mode in Ruby. Therefore, we can understand the Worker in Sidekiq in anthropomorphic ways. I own a Worker, but one thing to note is that these workers execute tasks according to the "Operation Manual", so they are not restricted to a certain type of tasks.

Queue

The significance of Queue is to differentiate tasks and Queue tasks. In Sidekiq, each type of tasks is separated by a Queue.

Redis Server

It refers to the Redis source for storing tasks. In the Sidekiq 2.x era, there is a bottleneck that only one Redis Server can be owned regardless of the number of Sidekiq instances, that is, the maximum task processing speed is limited to the processing speed of a single Redis server per second, which is about 5000 job/s. However, after Sidekiq 3.0, it extends the redis_pool parameter, each Worker can choose to use Redis Server.

Redis Client

As a task submitter, Redis submits tasks to the specified Redis Client through Worker.

Development Environment ubuntu 12.4, ruby 2.12, rails 4.1.1, redis 3.0.7, sidekiq 3.1.3 installation and configuration

• Add Gemfile:

Gem "redis", "~> 3.0.7 "gem 'sidekiq 'gem 'sinatra '# Use the built-in monitoring page

 

• Create the sidekiq. rb file under initializers to initialize Redis and Sidekiq config. Initializers/sidekiq. rb:

Redis_server = '2017. 0.0.1 '# redis server redis_port = 6379 # redis port redis_db_num = 0 # redis database No. redis_namespace = 'highlander22 _ sidekiq' # namespace, custom Sidekiq. configure_server do | config | p redis_server # Remove config. redis = {url: "redis: // # {redis_server }:#{ redis_port}/# {redis_db_num}", namespace: redis_namespace} end Sidekiq. configure_client do | config. redis = {url: "redis: // # {redis_server }:#{ redis_port}/# {redis_db_num}", namespace: redis_namespace} end

• Sidekiq start the configuration file config/sidekiq. yml:

: Concurrency: 5 # concurrency: pidfile: tmp/pids/sidekiq. pid: logfile :. /log/sidekiq. log # output log address: queues:-default # written in the queue parameter, which indicates that sidekiq is allowed to process this queue-[myqueue, 2] # written as an array, the first parameter is the name of the enabled queue, and the second parameter is the priority development: concurrency: 5 staging: concurrency: 10 production: concurrency: 20

 

• First, put the worker class in the app/workers folder, app/workers/hard_worker.rb:

class HardWorker  include Sidekiq::Worker   def perform(name, count)     # do somethings     puts 'Doing hard work'  endend

 

• Call HardWorker. cmdm_async in the Controller action or model:

HardWorker.perform_async('bob', 5)

• After the sidekiq Configuration Parameter command is added with -- help, you can see its configuration parameters:

richard@richard:~/ipucc/blog$ bundle exec sidekiq --help2014-06-12T10:16:35Z 22651 TID-6dezc INFO: sidekiq [options]-c, --concurrency INT processor threads to use-d, --daemon Daemonize process-e, --environment ENV Application environment-g, --tag TAG Process tag for procline-i, --index INT unique process index on this machine-q, --queue QUEUE[,WEIGHT] Queues to process with optional weights-r, --require [PATH|DIR] Location of Rails application with workers or file to require-t, --timeout NUM Shutdown timeout-v, --verbose Print more verbose output-C, --config PATH path to YAML config file-L, --logfile PATH path to writable logfile-P, --pidfile PATH path to pidfile-V, --version Print version and exit-h, --help Show help

 

After configuring these items, you can test the HardWorker you just wrote.

First, start Sidekiq

You need to start it in the rails project directory

You can use linux cli to add parameters to start sidekiq:Bundle exec sidekiq-q queue_name_1, queue_name_2
You can also put these parameters in yml and start them using the-C parameter:Bundle exec sidekiq-C config/sidekiq. yml
You can also directly:SidekiqOrBundle exec sidekiq-e production
-r: Specify the custom worker to be introduced and related ruby code.
-C: Specifies the path of the configuration file. If the path of the configuration file is config/sidekiq. yml, this parameter can be ignored.
-e: Specifies the environment in which the current sidekiq runs. (control the configuration information used)

richard@richard:~/ipucc/blog$ bundle exec sidekiq --help2014-06-12T10:16:35Z 22651 TID-6dezc INFO: sidekiq [options]-c, --concurrency INT processor threads to use-d, --daemon Daemonize process-e, --environment ENV Application environment-g, --tag TAG Process tag for procline-i, --index INT unique process index on this machine-q, --queue QUEUE[,WEIGHT] Queues to process with optional weights-r, --require [PATH|DIR] Location of Rails application with workers or file to require-t, --timeout NUM Shutdown timeout-v, --verbose Print more verbose output-C, --config PATH path to YAML config file-L, --logfile PATH path to writable logfile-P, --pidfile PATH path to pidfile-V, --version Print version and exit-h, --help Show help

 

Use the Rails Console for testing

Richard @ richard :~ /Ipucc/blog $ rails cLoading development environment (Rails 4.1.1) 2.1.2: 002> HardWorker. Handler m_async 'Hello world', 1 # Call perform => "handler" 2.1.2: 003>

Use the built-in monitoring page

Add route. rb:

require 'sidekiq/web'mount Sidekiq::Web => '/sidekiq'

Access http: // localhost: 3000/sidekiq/retries to enter the monitoring page

In this way, we can see the running status of sidekiq on the page. For more details, see sidekiq Monitoring.