PYTHON+PIKA+RABBITMQ environment deployment and implementation Work queue

RABBITMQ Chinese translation, the main is the MQ letter: Message queue, that is, the meaning of messages queuing. There is also a rabbit word, that is, the meaning of the rabbit, and Python language is called Python, the foreigner is quite humorous. RABBITMQ services are similar to MySQL, Apache services, but offer different functions. RABBIMQ is used to provide a service for sending messages that can be used to communicate between different applications.

Installing RABBITMQ
To install the next rabbitmq, under Ubuntu 12.04 can be installed directly through the Apt-get:

sudo apt-get install Rabbitmq-server

After the installation, the RABBITMQ service has been started. Next look at Python's example of writing Hello world!. The content of the instance is to send "Hello world!" from send.py To rabbitmq,receive.py receive send.py sent from RABBITMQ.

where P denotes produce, producer's meaning, also can be called sender, Examples show that send.py;c represents consumer, the meaning of the consumer, also known as the receiver, the instance is receive.py, the middle red indicates the meaning of the queue, and the instance shows the Hello queue.

Python uses the RABBITMQ service and can use a ready-made class library Pika, TXAMQP, or Py-amqplib, where Pika is selected.

Installing Pika

Install Pika can be installed using PIP, Pip is a Python software Management Pack, if not installed, can be installed through Apt-get

sudo apt-get install Python-pip

Install Pika with PIP:

sudo pip install Pika

send.py Code

Connect to the RABBITMQ server because it is tested locally, so use localhost.

Connection = Pika. Blockingconnection (Pika. Connectionparameters (        ' localhost ')) channel = Connection.channel ()

Declares the message queue in which the message is passed. If a message is sent to a queue that does not exist, RABBITMQ will automatically clear the messages.

Channel.queue_declare (queue= ' Hello ')

Send a message to the Hello queue declared above, where exchange represents the exchanger, can specify exactly which queue the message should be sent to, Routing_key is set to the name of the queue, the body is the content sent, the details of the delivery is not the first attention.

Channel.basic_publish (exchange= ", routing_key= ' hello ', body= ' Hello world! ')

Close connection

Connection.close ()

Full code

#!/usr/bin/env python#coding=utf8import Pika connection = Pika. Blockingconnection (Pika. Connectionparameters (        ' localhost ')) channel = Connection.channel () channel.queue_declare (queue= ' Hello ') Channel.basic_publish (exchange= ", routing_key= ' hello ', body= ' Hello world! ') print "[x] Sent ' Hello world! '" Connection.close ()

First to execute the next program, successful execution, RABBITMQCTL should successfully increase the Hello queue, and there should be a message in the queue, with the Rabbitmqctl command to view the next

Rabbitmqctl list_queues

On the author's computer output The following information:

There is indeed a Hello queue, and there is a message in the queue. Next, use receive.py to get the information in the queue.

receive.py Code

As in the previous two steps of send.py, the first thing to do is to connect to the server and then declare the message queue, where the same code is no longer affixed.

Receiving a message is more complex, you need to define a callback function to handle, the callback function here is to print out the information.

DEF callback (ch, method, properties, body):  print "Received%r"% (body,)

Tell RABBITMQ to use callback to receive information

Channel.basic_consume (callback, queue= ' Hello ', no_ack=true)

Begins receiving information and enters a blocking state, and information in the queue is called callback for processing. Press CTRL + C to exit.

Channel.start_consuming ()

Full code

#!/usr/bin/env python#coding=utf8import Pika connection = Pika. Blockingconnection (Pika. Connectionparameters (        ' localhost ')) channel = Connection.channel () channel.queue_declare (queue= ' Hello ') def Callback (ch, method, properties, body):  print "[x] Received%r"% (body,) Channel.basic_consume (callback, queue= ' Hel Lo ', no_ack=true) print ' [*] waiting for messages. To exit Press CTRL + C ' channel.start_consuming ()

Executes the program and is able to receive the message Hello world! in the queue hello, and then print it on the screen. Change a terminal, execute send.py again, you can see receive.py this side will receive the message again.

Work Queue Example

1. Preparatory work (preparation)

In the instance program, the new_task.py is used to simulate the task assignment, and worker.py to simulate the worker.

Modify send.py, receive information from command line arguments, and send

Import SYS message = '. Join (sys.argv[1:]) or "Hello world!" Channel.basic_publish (exchange= ',           routing_key= ' Hello ',           body=message) print "[x] Sent%r"% (message,)

Modify the callback function of the receive.py.

Import Time DEF callback (ch, method, properties, body):  print "[x] Received%r"% (body,)  Time.sleep (Body.count ( '.') )  print "[x] Done"

This side first open two terminals, all running worker.py, in the listening state, this is equivalent to two workers. Open a third terminal, run new_task.py

$ python new_task.py first message.$ python new_task.py Second message: $ python new_task.py third message...$ python new_task.py fourth message....$ python new_task.py fifth message .....

Observing worker.py received the task, one of the workers received 3 tasks:

$ Python worker.py [*] waiting for messages. To exit Press CTRL + C [x] Received ' first message. ' [x] Received ' third message ... ' [x] Received ' fifth message ... '

In addition a worker receives 2 tasks:

$ Python worker.py [*] waiting for messages. To exit Press CTRL + C [x] Received ' Second message: ' [x] Received ' fourth message .... '

From above, each worker is assigned to a task in turn. So if a worker hangs out when he is working on a task, the task is not completed and should be dealt with by other workers. So there should be a mechanism to give feedback when a worker finishes a task.

2. Confirmation of messages (message acknowledgment)

The message confirms that when the worker finishes the task, it feeds back to RABBITMQ. To modify a callback function in worker.py:

DEF callback (ch, method, properties, body):  print "[x] Received%r"% (body,)  Time.sleep (5)  print "[x] Done"  ch.basic_ack (Delivery_tag = Method.delivery_tag)

This side pauses for 5 seconds, can convenient CTRL + C exit.

It is also possible to remove the No_ack=true parameter or set it to false.

Channel.basic_consume (callback, queue= ' Hello ', no_ack=false)

With this code, even if one of the workers exits with CTRL + C, the task being performed is not lost, RABBITMQ will reassign the task to another worker.

3. Message Persistence Store (msg durability)

Although there is a message feedback mechanism, if RABBITMQ itself hangs, then the task will still be lost. So you need to persist the task to store it. Declaring persistent storage:

Channel.queue_declare (queue= ' Hello ', durable=true)

However, this program executes an error because Hello is already present and is non-persistent, and RABBITMQ does not allow the use of different parameters to redefine the existing queue. Redefine a queue:

Channel.queue_declare (queue= ' task_queue ', durable=true)

When sending a task, use delivery_mode=2 to mark the task as persistent storage:

Channel.basic_publish (exchange= ",           routing_key=" Task_queue ",           body=message,           Properties=pika. Basicproperties (             Delivery_mode = 2, # make message persistent           ))

4. Fair Dispatch (Fair Dispatch)

In the above example, each worker is assigned to a task sequentially, but each task is not necessarily the same. Possible tasks are heavier, execution time is relatively long, and some tasks are relatively light, the execution time is relatively short. If it is best to dispatch fairly, use Basic_qos to set up prefetch_count=1 so that RABBITMQ does not assign multiple tasks to the worker at the same time, that is, the task will not be received again until the worker has completed the task.

Channel.basic_qos (Prefetch_count=1)

new_task.py Complete code

#!/usr/bin/env pythonimport pikaimport SYS connection = Pika. Blockingconnection (Pika. Connectionparameters (host= ' localhost ')) channel = Connection.channel () channel.queue_declare (queue= ' Task_queue ', durable=true) message = '. Join (sys.argv[1:]) or "Hello world!" Channel.basic_publish (exchange= ', routing_key= ' task_queue ', Body=message, properties=pika.b Asicproperties (Delivery_mode = 2, # make message persistent)) print "[x] Sent%r"% (message,) conn Ection.close () worker.py full code #!/usr/bin/env pythonimport pikaimport Time connection = Pika. Blockingconnection (Pika. Connectionparameters (host= ' localhost ')) channel = Connection.channel () channel.queue_declare (queue= ' Task_queue ', durable=true) print ' [*] waiting for messages. To exit Press CTRL + C ' def callback (ch, method, properties, body): print "[x] Received%r"% (body,) Time.sleep (body.co  Unt ('. ')) print "[x] Done" ch.basic_ack (Delivery_tag = Method.delivery_tag) channel.basic_qos (prefetch_count=1) Channel.basic_consume (callback, queue= ' Task_queue ') channel.start_consuming ()