Linear optimizations in Python, part 2nd building an extensible infrastructure in the cloud

Brief introduction

The 1th installment of this three-part series describes the basics of using the Pyomo Library for linear optimization in Python. Now we'll explain how to extend it. How does Python extend it without a real operating system thread? This article will show you how to combine these technologies to create an actual, extensible infrastructure that can be used to build a Pyomo Web solution. We use a single threaded event loop, a AMQP server, and a worker thread process to create a pattern that expands a linear optimization system using this pattern. The infrastructure also applies to many common computing problems in Python or Ruby.

Extended

The challenge of creating extensible, parallel-running code in scripting languages like Python is different from the C, C + + or Java programming languages. Two important factors are execution speed and line Cheng. More complex, Amdahl's law shows that the acceleration of a program is not strictly proportional to the number of processors available on the computer. The scale of the code-side portion is ultimately limited by the time required for the sequential part of the program.

An intuitive way to consider this problem is to think of the problem as an event loop in which each tick (tick) of the event loop takes 1 seconds of CPU time. Within the event loop, if you try to debug, you will find that 75% of the time is spent on the time.sleep (. 75) command, and the rest of the time is spent doing the work. Because 75% of the time of this one-second event cycle is spent on hibernation, it is impossible to run the part of the loop in parallel. In this second time, the only thing that can run faster is the 25% part that is not dormant. In this. 25-second event loop, no matter how fast the processor runs or how many processors are thrown, it always takes. More than 75 seconds to optimize the "can parallel" code.

Even with the Amdahl law, adding more threads will not, in principle, paralyze the parallel parts of the Python code. Although such strategies are very useful in languages such as C # and Java, they are slightly more complex in Python. In Python, if part of the code can be parallelization, but this part of the code assumes CPU work, the thread does not provide any optimization assistance because of the Global interpreter Lock (GIL).

The last complication is that if you write a WEB application that uses an event-driven infrastructure such as Tornado to perform linear optimizations, you must incorporate some special ideas into your design. If the event loop is blocked and cannot perform CPU work or block network operations, then the design can safely invoke the iteration server. W. Richard Stevens describes two types of servers in the Unix network programming Book:

Iteration Server: Unable to process the pending client until the current client service is completely terminated.

Concurrent servers: Typically, a child process is generated for each request.

If not carefully considered, the event loop server (Ruby's event Machine and Python's Tornado) will soon become an iterative server. In the real world, an iterative server is a toy that must be avoided at all costs.

Event Loop Programming

For an event loop, CPU-intensive work that performs a linear optimization algorithm locks all requests made by the customer until the first requested worker process completes. In many cases, this is not a best practice because other CPU-intensive work can be done using the other cores on the server. One of the things that can easily happen is that 23 of the 24 cores in one server do nothing, while CPU-intensive requests start to form an exponential queue. Similarly, if there is a blockage of the network in the event loop, there may be a worse scenario-24 cores are idle and the request is queued.

The trick to using an event-cycling infrastructure is to ensure two things:

All network operations must be done in a non-blocking manner, either in a separate thread or process, either using I/O multiplexing technology (such as selection and polling), or using asynchronous I/O to the POSIX aio_ function.

In the event loop, the CPU or network operation should be minimized because it blocks the event loop.

In short, if you use it as a Web site, there is no blocking event loops. Other steps should be taken to get the CPU to do other work. In Python, a policy is to receive messages synchronously, and then transfer the task load to a series of worker threads subscribing to the Advanced Message Queueing Protocol (AMQP) messaging bus (for example, RABBITMQ). This is the strategy that this article will introduce. The best balance point of an asynchronous event loop server is to handle many concurrent, long-running socket connections. A real example is sending data to another connection (such as a websockets connection) from a network connection, such as a socket that collects stock data.

Create an Tornado Web application to distribute Pyomo applications

The example uses several different techniques:

Tornado is an open source, extensible, non-blocking WEB server. It can act as an asynchronous messaging system.

The linear optimization task that the Pyomo service runs on each request.

RABBITMQ can act as a message bus between layers.

To build the environment and run the instance, you need to install the following components on OS X.

Use HomeBrew to install RABBITMQ.

Pyomo

Pika, a rabbitmq Python client.

Tornado, an asynchronous Web server. Use Easy_install installation.