Python Twisted Learning Series 20 (reprinted stulife The best Twisted Introductory tutorial)

Wheels in part 20th: Introduction to Twisted and Erlang

In this series, there is a fact that we have not introduced, that is, hybrid synchronous "plain python" code and Asynchronous twisted code is not a simple task, because blocking the indefinite time in the twisted program will lose the benefits of the asynchronous model.

If you are in the first contact with asynchronous programming, then your knowledge seems to have some limitations. You can use these new technologies within the twisted framework, rather than in the broader world of generic Python code. At the same time, when working with twisted, You are confined to only those libraries that are written specifically as part of the twisted program, at least if you want to call them directly from the reactor thread.

But asynchronous programming has been around for years and is barely limited to twisted. In fact, there is an astonishing number of asynchronous programming models in Python. A search will see a lot. They differ in detail from twisted, but basic ideas (such as asynchronous I/O, splitting large-scale data streams into small chunks) are the same. So if you need, or choose, to use a different framework, you will have a good start by learning twisted.

As we move beyond Python, we also find that many languages and systems are either based on or using the asynchronous programming model. The knowledge you have learned in twisted will continue to serve you in developing a broader field of asynchronous programming.

In this section, we will take a brief look at Erlang, a programming language, and a runtime system, which widely uses asynchronous programming concepts, but in a unique way. Please note that we are not going to start writing Erlang. Instead, explore some of the ideas contained in Erlang, Look at these links to twisted thoughts. The basic theme is that the knowledge you get from learning twisted can be applied to other techniques.

Recalling callbacks

Consider Figure 6, a graphical representation of the callback. is:d oc: ' p06 ' introduced in the verse Proxy 3.0 callback and datareceived method in the order of the poetry client principle. A callback is fired every time a small portion of a poem is downloaded from a connected poetry server.

Suppose our client downloads 3 poems from 3 different servers. Look at the problem from a reactor point of view (which is always advocated in this series), we get a single big loop that fires one or more callbacks at each turn, 40:

Figure 40:reactorThe callback of the angle

This figure shows that reactor is working merrily, and every time the poem arrives it calls datareceived. Each datareceived call is applied to a specific poetryprotocal class instance. We know a total of 3 instances because we are downloading 3 poems (so there must be 3 connections).

Consider this diagram in the perspective of a protocol instance. Remember that each protocol has only one connection (a poem). The instance can "see" a method call flow, each of which carries the next part of the poem, as follows:

DataReceived (self, "when I had fears") datareceived (self, ' that I could cease to be ') datareceived (self, "before my pen has Glea ") datareceived (self," n ' d my teeming brain ") ...

However, this is not strictly a Python loop, and we can turn it into a loop:

For data in Poetry_stream (): # pseudo-code    datareceived (data)

We can imagine a "callback loop", 41:

Figure 41: A virtual callback loop

Again, this is not A for loop or while loop. The only important Python loop in our poetry client is reactor. But we can see each protocol as a virtual cycle, and when there is poetry it will start the loop. Based on this idea, we can refactor the entire client in Figure 42:

Figure 42:reactorTurning virtual loops

In this picture, there is a cycle-- reactor and three virtual loops--The individual of the poem agreement instance. Cycle turned up, so that the virtual cycle also turned, like a set of interlocking gears.

Enter Erlang

Erlang, like Python, derives from a programming language of a general purpose dynamic type created in the 80 's. Unlike Python, Erlang is functional rather than object-oriented, and is syntactically similar to a nostalgic Prolog, which was originally implemented by Erlang. Erlang is designed to build a highly reliable distributed telephony system, so Erlang includes extensive network support.

One of the most unique features of Erlang is a concurrency model that involves lightweight processes. An Erlang process is neither an operating system process nor a thread. It is a function that runs independently in the Erlang runtime environment, and it has its own stack. The Erlang process is not a lightweight thread because the Erlang process cannot share state (many data types are also immutable, and Erlang is a functional programming language). An Erlang process can interact with other Erlang processes, but only by sending messages, the message is always, At least conceptually, it is replicated instead of shared.

So an Erlang program looks 43:

Figure 43: Erlang program with 3 processes

In this diagram, the individual process becomes "real". Because the process is the first construct in Erlang, it is like an object in Python. But the runtime becomes "virtual", not because it doesn't exist, but because it's not a simple loop. The Erlang runtime can be multithreaded because it has to implement a comprehensive programming language and is responsible for many things other than asynchronous I/O. Further, a language runtime is also a medium that allows Erlang processes and code execution, rather than reactor in twisted that extra structure.

So a better representation of an Erlang program such as 44:

Figure 44: Erlang program with several processes

Of course, the Erlang runtime does need to use asynchronous I/O and one or more selection loops, because Erlang allows you to create a large number of processes. Large-scale Erlang programs can start thousands of erlang processes, so assigning an actual OS thread to each process is the problem. If Erlang allows multiple processes to perform I/O while allowing other processes to run even if that I/O is blocked, then asynchronous I/O must be included.

Note Our illustration of the Erlang program shows that each process is "running by its own power" rather than being rotated by a callback. As reactor 's work is summed up as the structure of the Erlang runtime, callbacks no longer play a central role. The problem that needs to be resolved by callbacks in twisted is resolved by sending an asynchronous message from one process to another in Erlang.

An Erlang poetry agency

Let's take a look at the Erlang poetry client. This time we jump directly into the work version instead of building it slowly in twisted. Again, this is not meant to be the full version of Erlang introduction. But if this arouses your interest, we have finally suggested some deep readings in this section.

The Erlang client is located in Erlang-client-1/get-poetry. In order to run it, of course you need to install Erlang. The following code isMainfunction code, as in the Python client.MainFunctions have the same purpose:

Main ([]),    usage (), main (args)    , Addresses = Parse_args (args),    main = self (),    [Erlang:spawn_ Monitor (Fun () get_poetry (Tasknum, Addr, Main) end)                | | {tasknum, Addr} <-Enumerate (Addresses)],        collect_poems (Length (Addresses), []).

If you have never seen a prolog or similar language, then the syntax of Erlang will be a bit odd. But some people think so too.MainThe function is defined by two separate groups of sentences, separated by semicolons. Erlang chooses which sentence group to run according to the parameters, so the first sentence group runs only when we execute the client without providing any command-line arguments, and it only prints out the help information. The second group of sentences is all practical action.

Each statement in the Erlang function is separated by commas, so the function ends with a period. Let's take a look at the second group, where the first line parses only the command-line arguments and binds them to a variable (all variables in Erlang must be capitalized). The second line uses self function to get the ID of the currently running Erlang process (not the OS process). Since this is the main function, you can think of it as equivalent to the __main__ module in Python. The third line is the most interesting:

[Erlang:spawn_monitor (Fun (), Get_poetry (Tasknum, Addr, Main) end)     | | {tasknum, Addr} <-Enumerate (Addresses)],

This statement is an understanding of the Erlang list and has a similar syntax to Python. It produces a new Erlang process that corresponds to each server that needs to be connected. At the same time, each process will run the same get_poetry function, but with different parameters depending on the particular server. We pass the PID of the main process at the same time so that the new process can send the poem back (you usually need a process PID to send it a message)

The last statement in the main function calls the collect_poems function, which waits for the verse to pass back and the get_poetry process to end. We can look at other functions, but first you might compare the main of Erlang The function is equivalent to twisted the main function in the client.

Now let's take a look at the Get_poetry function in Erlang. In fact, there are two functions in our script called Get_poetry. In Erlang, a function is determined by the name and the number of tuples, so our script contains two different functions, GET_POETRY/3 and GET_POETRY/4, they receive 3 or 4 parameters respectively. Here is GET_POETRY/3, which is generated by main :

Get_poetry (Tasknum, Addr, Main),    {host, port} = Addr,    {OK, Socket} = Gen_tcp:connect (host, Port,                                   [ Binary, {active, false}, {packet, 0}]),    get_poetry (Tasknum, Socket, Main, []).

This function first makes a TCP connection, just like the get_poetry in the twisted client. But then, instead of returning, instead of continuing to use that TCP connection, by calling GET_POETRY/4, the following:

Get_poetry (tasknum, Socket, Main, Packets), Case    gen_tcp:recv (socket, 0)        of {OK, Packet}            io: Format ("Task ~w:got ~w bytes of poetry from ~s\n",                      [Tasknum, Size (Packet), PeerName (Socket)]),            get_poetry ( Tasknum, Socket, Main, [packet| Packets]);        {error, _},            Main! {poem, list_to_binary (Lists:reverse (Packets))}    End.

This Erlang function is doing the work of Poetryprotocol in the twisted client, but the difference is that it uses blocking function calls. The GEN_TCP:RECV function waits for the arrival of some data on the socket (or the socket to close), regardless of how long it takes. But the "blocking" function in Erlang only blocks the process that is running the function, not the entire Erlang runtime. The TCP socket is not a true blocking socket (you cannot create a real blocking socket in the pure Erlang code). For each socket in Erlang, somewhere in the runtime, a "real" The TCP socket is set as a non-blocking model and is used as part of the selection cycle.

But the Erlang process does not know that. It just waits for some data to come in, and if it blocks, the other Erlang processes will run instead. Even a process never blocks, and the Erlang runtime can freely switch between processes at any time. In other words, Erlang has a non-synergistic concurrency mechanism.

Note GET_POETRY/4, after receiving a small portion of the poem, continues to invoke itself recursively. For an urgent language programmer this looks like a recipe for draining memory, but the Erlang compiler can optimize the "tail" call (the function calls the last statement in a function) As a loop. This illuminates another interesting parallel between the Erlang client and the twisted client. In the twisted client, the "virtual" loop is created by Reaactor , which invokes the same function again and again ( DataReceived). In the Erlang client at the same time, the "real" running process (GET_POETRY/4) is formed by "tail tuning" to call their own loops again and again. How do you feel?

If the connection is closed, the last thing Get_poetry do is send the poem to the main process. Also end the process that Get_poetry is running, because there's nothing left to do.

The remaining key functions in our Erlang client are collect_poems:

Collect_poems (0, poems)    , [Io:format ("~s\n", [P]) | | P <-Poems];collect_poems (N, Poems),    receive        {' Down ', _, _, _, _},            Collect_poems (N-1, Poems); C6/>{poem, poem},            Collect_poems (N, [poem| Poems])    end.

This function is run by the main process, like Get_poetry, which recursively loops over itself. It also blocks. The receive tells the process to wait for a message that conforms to the given pattern and extracts the message from the mailbox.

The Collect_poems function waits for two messages: Poetry and "Down" notifications. The latter is sent to the main process when one of the get_poetry processes is dead for some reason (this is the monitoring part of the Spawn_monitor). With a few down messages, We know when all the poetry is over. The former is a message from the Get_poetry process that contains the complete poem.

OK, let's run the Erlang client. Start with 3 slow servers first:

Python blocking-server/slowpoetry.py--port 10001 poetry/fascination.txtpython blocking-server/slowpoetry.py--port 10002 poetry/science.txtpython blocking-server/slowpoetry.py--port 10003 poetry/ecstasy.txt--num-bytes 30

Now we can run the Erlang client, which has a similar command-line syntax to the Python client. If you're on a Linux or other unix--like system, you should be able to run the client directly (assuming you've installed Erlang and made it on your path). In Windows, You may need to run the Escript program, which points to the Erlang client's path as the first parameter (the other parameters are left to the parameters of the Erlang client itself).

./erlang-client-1/get-poetry 10001 10002 10003

After that, you can see the following output:

Task 3:got bytes of poetry from 127:0:0:1:10003task 2:got bytes of poetry from 127:0:0:1:10002task 1:got of poetry from 127:0:0:1:10001 ...

This is like one of the previous Python clients that prints the information we get from each part of the poem. When all the poems are finished, the client should print the full contents of each poem. Note that the client switches between all servers, depending on which server can send poetry.

Figure 45 shows the process structure of the Erlang client:

Figure 45:erlang Poetry Client

This figure shows 3 Get_poetry processes (one per server) and one master process. You can see the message flow from the poetry process to the main process.

So what happens when a server fails? Let's try:

./erlang-client-1/get-poetry 10001 10005

The above command contains an active port (assuming you did not terminate the previous poetry server) and an inactive port (assuming you did not run either server on Port 10005). We get the following output:

Task 1:got bytes of poetry from 127:0:0:1:10001=error report==== 25-sep-2010::21:02:10 ===error in Process <0.33.0& Gt With exit Value: {{badmatch,{error,econnrefused}},[{erl_eval,expr,3}]}task 1:got-bytes of poetry from 127:0:0:1:10001 Task 1:got bytes of poetry from 127:0:0:1:10001 ...

The end client finishes the poem download from the Active server, prints out the poem and exits. So how does the main function know that the two processes are working? The error message is a clue. This error originates when Get_poetry attempts to connect to the server without getting the expected value ({OK, Socket}), but instead gets a connection denied error.

An unhandled exception in the Erlang process causes it to "crash", which means that the process stops running and all of their resources are recycled. But the master process, which monitors all get_poetry processes, receives a down message when any process stops running for whatever reason. This way, Our client has exited instead of running forever.

Discuss

Let's summarize the features of the twisted and Erlang clients about parallelism:

1. They are all connected to all poetry servers at the same time (or try to connect).
2. They are all instantly receiving poetry from the server, regardless of which server is sending it.
3. They all deal with poetry in a small way, so it is necessary to preserve part of the poetry that has been received so far.
4. They all create an "object" (or Python object or Erlang process) to handle all work for a particular server.
5. They all need to be careful to determine when a poem ends, regardless of whether a particular download succeeds or not.

In the end, the main function in two clients receives poems and "task completion" notifications asynchronously. In the twisted client this information is sent through Deferred , while in Erlang the client receives messages from the internal process.

Notice that two clients are very much alike, regardless of their overall policy or code schema. But the mechanism is a little different, one is to use objects, deferreds and callbacks, and the other is to use processes and messages. However, in the high-level thinking model, two clients are very similar, If you are familiar with both languages it is convenient to convert one to another.

Even the reactor mode is reproduced in a miniaturized form in the Erlang client. Our poetry. Each Erlang process in the client is eventually transformed into a recursive loop:

1. Wait for something to happen (a few verses come, a poem passes, another process ends), and
2. Take the appropriate action.

You can think of the Erlang program as a collection of small reactor , each rotating itself and occasionally sending a message to another small reactor (it will process this information with another event).

In addition, if you go deeper into Erlang, you'll see the callbacks appear. Erlang's gen_server process is a generic reactor loop, and you can "instantiate" it with a series of callback functions, which is a recurring pattern in the Erlang system.

Further reading

In this section we focus on the similarities between twisted and Erlang, but they are a lot different. One of the unique features of Erlang is the way it handles errors. A large Erlang program is structured as a tree-structured process group, with "regulators" on the upper level and "workers" on the leaves. If a worker process crashes, the regulatory process will notice and act accordingly ( Usually restarts the failed process).

If you're interested in learning Erlang, then you're lucky. Many of the books on Erlang have been published or will be published:

• Programming erlang--Author is one of the inventors of Erlang. A fascinating introduction to this language.
• Erlang programming--This book complements Armstrong's book and further details in many key sections.
• Erlang and OTP in action--this book has not yet been published, but I am waiting. Neither of the first two books introduces the OTP and constructs the Erlang framework for large applications. Full disclosure: The two authors of this book are My friends.

So much for Erlang first. In the next section we'll look at Haskell, another functional language, but it's different from Python and Erlang. However, we will try to find something in common.

Python Twisted Learning Series 20 (reprinted stulife The best Twisted Introductory tutorial)