Implementation of MQ Message Queuing for Python threads and introduction of advantages and disadvantages

Message Queuing (Mq,message queue) in the message data transmission of the preservation of data communication provides a guarantee and real-time processing convenience, here we take a look at the python thread of the MQ Message Queuing implementation and the advantages of Message Queuing analysis

Message Queuing is the container in which messages are saved during the transmission of a message. The Message Queue Manager acts as an intermediary when relaying a message from its source to its destination. The primary purpose of the queue is to provide routing and guarantee the delivery of messages, and if the recipient is unavailable when the message is sent, Message Queuing retains the message until it can be successfully passed. Believe that Message Queuing is a critical component for any architecture or application, here are 10 reasons:

Example of a Python message queue:

1.threading+queue Implementing thread Queues

#!/usr/bin/env python import queueimport threadingimport time queue = Queue.queue () class Threadnum (threading. Thread): "" "does not print a number wait 1 seconds, how many seconds to print 10 digits concurrently? "" "Def __init__ (self, queue):  Threading. Thread.__init__ (self)  self.queue = Queue  def run (self):  whiletrue:   #消费者端, getting num num from the queue   = Self.queue.get ()   print "I m num%s"% (num)   time.sleep (1)   #在完成这项工作之后, using Queue.task_done () The function sends a signal to the queue that the task has completed   self.queue.task_done () start = Time.time () def main (): #产生一个 The threads pool and passes the message to the thread function for processing, This opens 10 concurrent for I in range:  t = threadnum (queue)  T.setdaemon (True)  T.start ()   #往队列中填错数据  For num in range:   queue.put (num) #wait on the queue until everything have been processed queue.join () main () print "Elapsed time:%s"% (Time.time ()-start)

Operation Result:

I ' m num 0i ' m num 1i ' m num 2i ' m num 3i ' m num 4i ' m num 5i ' m num 6i ' m num 7i ' m num 8i ' m num 9Elapsed time:1.01399993896

Interpretation:

The specific work steps are described below:

1, create an instance of Queue.queue (), and populate it with data.

2, the instance of the populated data is passed to the thread class, which is inherited by the threading. Thread is created in the way.

3, generate the daemon thread pool.

4, each time a project is fetched from the queue, and the data in that thread and the Run method are used to perform the appropriate work.

5, after completing this work, use the Queue.task_done () function to send a signal to the queue that the task has completed.

6, performing a join operation on a queue actually means waiting until the queue is empty before exiting the main program.

One thing to note when using this pattern is that the program runs out of auto-exits by setting the daemon to true. The advantage is that you can perform a join operation on the queue before exiting, or wait until the queue is empty.

2. Multiple queues

So-called multiple queues, the output of one queue can be entered as another queue

#!/usr/bin/env pythonimport queueimport threadingimport Time queue = Queue.queue () Out_queue = Queue.queue () class ThreadN Um (threading. Thread): Def __init__ (self, queue, Out_queue): Threading.      Thread.__init__ (self) self.queue = Queue Self.out_queue = Out_queue def run (self): whiletrue: #从队列中取消息 num = Self.queue.get () bkeep = num #将bkeep放入队列中 self.out_queue.put (bkeep) #signals to queue J OB is the done Self.queue.task_done () class Printlove (threading. Thread): Def __init__ (self, out_queue): Threading. Thread.__init__ (self) self.out_queue = Out_queue def run (self): whiletrue: #从队列中获取消息并赋值给bkeep bkeep = SE       Lf.out_queue.get () Keke = "I Love" + str (bkeep) print Keke, print self.getname () time.sleep (1)  #signals to queue job is do Self.out_queue.task_done () start = Time.time () def main (): #populate queue with data For num in range: queue.put (num) #spawn a pool of threads, andPass them queue instance for I in range (5): T = threadnum (queue, Out_queue) T.setdaemon (True) T.start () for I In range (5): pl = Printlove (out_queue) Pl.setdaemon (True) Pl.start () #wait on the queue until everything have B Een processed queue.join () Out_queue.join () main () print "Elapsed time:%s"% (Time.time ()-start)

Operation Result:

I Love 0 thread-6i Love 1 thread-7i Love 2 thread-8i Love 3 thread-9i Love 4 thread-10i Love 5 thread-7i Love 6 thread-6i Love 7 thread-9i Love 8 thread-8i Love 9 thread-10elapsed time:2.00300002098

Interpretation:

Threadnum class Workflow

Define Queue---> Inherit threading----> Initialize Queue----> Define the data in the Run function--->get queue----> Process data---->put data to another queue-- > signaled to queue that the article was processed

Main function Workflow:

---> Throw data into a custom queue

--->for loop determines the number of threads started----> Instantiate threadnum class----> Start thread and set daemon

--->for loop determines the number of threads started----> Instantiate Printlove class---> Start thread and set as Daemon

---> Wait for the message in the queue to finish executing the join. The main program is exited.

After understanding the approximate implementation of MQ, let's summarize the benefits of Message Queuing:

1. Decoupling

It is extremely difficult to anticipate what needs to be encountered in future projects at the start of the project. Message Queuing Inserts an implicit, data-based interface layer in the middle of processing, which is implemented on both sides of the process. This allows you to independently extend or modify the processing on both sides, as long as you ensure that they adhere to the same interface constraints.

2. Redundancy

Sometimes the process fails when processing data. Unless the data is persisted, it will be lost forever. Message Queuing persists the data until it has been fully processed, bypassing the risk of data loss in this way. In the insert-get-delete paradigm used by many message queues, before removing a message from the queue, it is necessary that your process explicitly indicate that the message has been processed and that your data is safely saved until you have finished using it.

3. Extensibility

Because Message Queuing decouples your processing, it is easy to increase the number of messages queued and processed, as long as additional processing is required. No need to change the code, do not need to adjust parameters. Expansion is as simple as adjusting the power button.

4. Flexibility & Peak handling capability

When your app is on the home page of Hacker News, you'll find that access traffic climbs to an unusual level. Your app still needs to continue to function in the event of a surge in traffic, but such bursts are not common, and it is a huge waste to be ready to invest in resources that can handle this type of peak access. Using Message Queuing enables critical components to withstand increased access pressure, rather than crashing completely because of requests that exceed the load. Check out our blog post on peak handling capabilities for more information on this.

5. recoverability

When a part of the system fails, it does not affect the entire system. Message Queuing reduces the degree of coupling between processes, so even if a process that processes messages is hung up, messages queued to the queue can still be processed after the system resumes. This ability to allow retries or delays in processing requests is often the difference between a slightly inconvenienced user and a frustrated user.

6. Delivery Guarantee

The redundancy mechanism provided by Message Queuing guarantees that the message can be actually processed, as long as a process reads the queue. On this basis, IRONMQ provides a "one-time-only" guarantee. No matter how many processes are picking up data from a queue, each message can be processed only once. This is possible because getting a message just "booked" the message and temporarily moved it out of the queue. Unless the client explicitly indicates that the message has been processed, the message is put back in the queue and can be processed again after a configurable amount of time.

7. Ordering Guarantee

In many cases, the order of data processing is important. Message Queuing is inherently sorted and ensures that data is handled in a specific order. IRONMO guarantees that message pastes are processed in the Order of FIFO (first-in, in-place), so that the message is positioned in the queue to retrieve their location from the queue.

8. Buffering

In any important system, there will be elements that require different processing times. For example, loading a picture takes less time than applying a filter. Message Queuing helps perform the most efficient execution of a task through a buffer layer-the processing of the write queue is as fast as possible, rather than being constrained by the pre-processing that is read from the queue. This buffering helps to control and optimize the speed of the data flow through the system.

9. Understanding Data Flow

In a distributed system, it is a huge challenge to get a general impression of how long and why user actions will take. The message series is easily aided by the frequency of messages being processed to identify poorly performing processes or areas where data flows are not optimized.

10. Asynchronous communication

Many times, you don't want or need to process messages immediately. Message Queuing provides an asynchronous processing mechanism that allows you to put a message in a queue, but not immediately handle it. How many messages you want to put into the queue, and then deal with them when you're happy.