Nodejs practice-eventproxy module controls concurrency _ node. js

This article shares my experiences with nodejs and how to use the eventproxy module to control concurrency. If you are interested, refer to the following objectives:

Create a lesson4 project and write code in it.

The code entry is app. js, when the node app is called. js, it will output the title, link and first comment of all topics in the CNode (https://cnodejs.org/) Community homepage, in json format.

Output example:

[{"Title": "[announcement] Send recruitment posts to students to pay attention to here", "href": "http://cnodejs.org/topic/541ed2d05e28155f24676a12", "comment1": "ha ha ha "}, {"title": "Publish a JavaScript syntax highlighting plug-in under Sublime Text", "href": "http://cnodejs.org/topic/54207e2efffeb6de3d61f68f", "comment1": "sofa! "}]

Challenges

Based on the above objectives, the author of comment1 and his points in the cnode community are output.

Example:

[{"Title": "[announcement] Send recruitment posts to students to pay attention to here", "href": "http://cnodejs.org/topic/541ed2d05e28155f24676a12", "comment1": "ha ha ha ", "author1": "auser", "score1": 80},...]

Knowledge Point

Discover the beauty of Node. js callback hell

Learn how to use eventproxy to control concurrency

Course Content

This chapter is the most awesome part of Node. js-asynchronous concurrency content.

In the previous lesson, we introduced how to use superagent and cheerio to retrieve the homepage content. You only need to initiate an http get request. However, this time, we need to retrieve the first comment of each topic. This requires us to initiate a request for the link of each topic and use cheerio to retrieve the first comment.

CNode currently has 40 topics per page, so we need to initiate 1 + 40 requests to achieve our goal in this lesson.

The latter's 40 requests are initiated concurrently :), and there will be no multithreading or locks. The Node. js concurrency model is different from that of multithreading, throwing away those ideas. To be more specific, for example, the question of asynchronous, asynchronous, and single-thread concurrency in Node. js is not going to be discussed. For those who are interested in this regard, it is strongly recommended @ Park Ling's "nine light a deep Node. js": http://book.douban.com/subject/25768396.

Some friends with relatively high performance may have heard of concepts such as promise and generator. But I will only talk about callback, mainly because I personally only like callback.

This course requires three libraries: superagent cheerio eventproxy (https://github.com/JacksonTian/eventproxy)
Let's write this program step by step.

First, app. js should be long like this

Var eventproxy = require ('eventproxy'); var superagent = require ('superagent'); var cheerio = require ('cheerio '); // The url module is Node. js standard library // http://nodejs.org/api/url.htmlvar url = require ('url'); var cnodeUrl = 'https: // cnodejs.org/'{superagent.get (cnodeUrl ). end (function (err, res) {if (err) {return console. error (err);} var topicUrls = []; var $ = cheerio. load (res. text); // get all the links on the home page $ ('# topic_list. topic_title '). each (function (idx, element) {var $ element = $ (element); // $ element. attr ('href ') is/topic/542acd7d5d28233425538b04 // we use url. resolve to automatically infer the complete url, into the form of // https://cnodejs.org/topic/542acd7d5d28233425538b04 // For more information, see the http://nodejs.org/api/url.html#url_url_resolve_from_to example var href = url. resolve (cnodeUrl, $ element. attr ('href '); topicUrls. push (href) ;}); console. log (topicUrls );});

Run node app. js

The output is as follows:

OK. Now we have obtained the addresses of all the URLs. Next, we will crawl these addresses and it will be done. Node. js is so simple.
Before capturing the file, you must introduce the eventproxy library.

Students who have written asynchronous data using js should know that if you want to concurrently obtain data of two or three addresses asynchronously and use the data together after obtaining the data, you can maintain a counter by yourself.

First define a var count = 0, and then count ++ after each successful capture. If you want to capture data from three sources, because you do not know who will complete these asynchronous operations first, when the capture is successful, count = 3. When the value is true, use another function to continue the operation.
Eventproxy plays the role of this counter to help you manage whether or not these asynchronous operations are complete. After completing these operations, eventproxy automatically calls the processing functions you provide, the captured data is also transmitted as a parameter.
Assume that we do not use eventproxy or counter, the method for capturing three sources is as follows:

// Refer to jquery's $. get Method

$.get("http://data1_source", function (data1) { // something $.get("http://data2_source", function (data2) {  // something  $.get("http://data3_source", function (data3) {   // something   var html = fuck(data1, data2, data3);   render(html);  }); });});

You have written all the above Code. First obtain data1, then obtain data2, then obtain data3, and then fuck them for output.

However, we should also think that the data from these three sources can be obtained in parallel. The acquisition of data2 does not depend on the completion of data1, and data3 does not depend on data2.

Therefore, we use counters to write them as follows:

(function () { var count = 0; var result = {}; $.get('http://data1_source', function (data) {  result.data1 = data;  count++;  handle();  }); $.get('http://data2_source', function (data) {  result.data2 = data;  count++;  handle();  }); $.get('http://data3_source', function (data) {  result.data3 = data;  count++;  handle();  }); function handle() {  if (count === 3) {   var html = fuck(result.data1, result.data2, result.data3);   render(html);  } }})();

It's not ugly. I write code to make it look good.

If we use eventproxy, write it as follows:

var ep = new eventproxy();ep.all('data1_event', 'data2_event', 'data3_event', function (data1, data2, data3) { var html = fuck(data1, data2, data3); render(html);});$.get('http://data1_source', function (data) { ep.emit('data1_event', data); });$.get('http://data2_source', function (data) { ep.emit('data2_event', data); });$.get('http://data3_source', function (data) { ep.emit('data3_event', data); });

It looks much better, right? It's a high counter.

ep.all('data1_event', 'data2_event', 'data3_event', function (data1, data2, data3) {});

This sentence listens to three events, namely data‑event, data2_event, and data3_event. Each time a source data is captured, it passes through ep. emit () to tell ep that the XX event has been completed.

When three events are not completed at the same time, ep. emit () does not do anything after it is called. When all three events are completed, the callback function at the end is called to process them in a unified manner.

Eventproxy provides APIs required in many other scenarios, but the most common usage is the above:

First var ep = new eventproxy (); get an eventproxy instance.

Tell it what events you want to listen to and give it a callback function. Ep. all ('event1', 'event2', function (result1, result2 ){}).
When appropriate, ep. emit ('event _ name', eventData ).

The idea of processing asynchronous concurrency in eventproxy is like a goto statement in the Assembly, and the program logic jumps everywhere in the code. The Code has already been executed to 100 rows, and the callback function of 80 rows suddenly starts to work again. If your asynchronous logic is complex, after the 80-row function is completed, another 60-row function is activated. The concurrency and nesting problems have been solved, but the goto statements that have been eliminated for decades have returned.

As for this idea, I personally think it is still not bad. It seems quite clear to be familiar with it. But js this slag language would have been messy, what variables improve (http://www.cnblogs.com/damonlan/archive/2012/07/01/2553425.html) Ah, no main function ah, variable scope ah, the data type is often as simple as numbers, strings, hash, and arrays. This series of problems is not a problem.
The programming language is ugly.

Back to the topic, we have obtained a topicUrls array with a length of 40, which contains links to each topic. This means that we will issue 40 concurrent requests. We need to use the # after API of eventproxy.

Everyone learn this API on their own: https://github.com/JacksonTian/eventproxy#%E9%87%8D%E5%A4%8D%E5%BC%82%E6%AD%A5%E5%8D%8F%E4%BD%9C
I directly pasted the code.

// After topicUrls is obtained, // obtain an eventproxy instance var ep = new eventproxy (); // The command ep repeatedly listens to topicUrls. length (40 times here) 'topic _ html 'event and then action ep. after ('topic _ html ', topicUrls. length, function (topics) {// topics is an array containing 40 ep. the 40 pair IN emit ('topic _ html ', pair) // start action topics = topics. map (function (topicPair) {// jquery uses var topicUrl = topicPair [0]; var topicHtml = topicPair [1]; var $ = cheerio. load (topicHtml); return ({title: $ ('. topic_full_title '). text (). trim (), href: topicUrl, comment1: $ ('. reply_content '). eq (0 ). text (). trim (),}) ;}); console. log ('final: '); console. log (topics) ;}); topicUrls. forEach (function (topicUrl) {superagent. get (topicUrl ). end (function (err, res) {console. log ('fetch' + topicUrl + 'successful '); ep. emit ('topic _ html ', [topicUrl, res. text]) ;});

The output length is as follows:

For the complete code, see the app. js file in the lesson4 directory.

Summary

The eventproxy module introduced today is used to control concurrency. Sometimes we need to send N http requests at the same time, and then use the obtained data for subsequent processing, this module can be used to conveniently determine that all data has been obtained concurrently. The module can be used not only on the server but also on the client.