1. The ability to parse the page
2. Ability to operate the database (MySQL)
3. Crawl efficiency
4. Code Volume
When recommending a language, please describe the required class library or framework.
For example: Python+mysqldb+urllib2+re
PS: Actually I'm not very fond of using python (probably on the Windows platform for reasons that require a variety of transcoding characters, and multithreading seems to be very chicken.) )
Reply content:
Mainly look at your definition of "crawler" for what to do.
1, if is directed to crawl several pages, do some simple page parsing,
Crawl efficiency is not the core requirement, then what language difference is not big.
Of course, if the page structure is complex, the regular expression is very complex, especially with the support of the XPath class library/crawler library, it will be found that although the entry threshold is low, but extensibility, maintainability and so on are very bad. In this case, it is recommended to use some off-the-shelf crawlers, such as XPath, multithreaded support, or factors that must be considered.
2, if is directed crawl, and the main goal is to parse the content of JS dynamic generation
At this time, the page content is Js/ajax dynamic generation, with the ordinary request page--parsing method is not used, need to use a similar Firefox, Chrome browser JS engine to do dynamic parsing of the page's JS code.
In this case, it is recommended to consider CASPERJS+PHANTOMJS or SLIMERJS+PHANTOMJS, of course, such as selenium can also be considered.
3, if the crawler is involved in large-scale site crawling, efficiency, scalability, maintainability, etc. is a factor that must be considered
Large-scale crawler crawling involves many problems: multithreading concurrency, I/o mechanism, distributed crawling, message communication, weighing mechanism, task scheduling, and so on, this time the language and the use of the framework of the choice is of great significance.
PHP is not recommended for multi-threaded, asynchronous support.
NodeJS: Crawling to some vertical sites is possible, but because of the weak support for distributed crawling, message communication, etc., judging by your own situation.
Python: It is highly recommended to support these issues. Especially the Scrapy framework is worth being the first choice. Many advantages: Support XPath, based on twisted, good performance, good debugging tools;
In this case, if you still need to do the parsing of JS dynamic content, Casperjs is not suitable, only based on such as Chrome V8 engine and other self-made JS engine.
As for C, C + + Although the performance is good, but not recommended, especially considering the cost and many other factors; For most companies it is recommended to do it based on some open source framework, do not invent the wheel yourself, make a simple crawler easy, but to make a complete crawler is very difficult.
Like I set up the website of the aggregation of the public number content / http lewuxian.com
is based on scrapy, and of course, Message Queuing. can refer to:
The specific content can refer to a task Scheduling distribution service Architecture
Let's talk a little bit about how I feel, PHP doesn't, I use Python and node. js.
Simple directional Crawl:
Python + urlib2 + RegExp + BS4
Or
node. js + Co, any one by one DOM frame or HTML parser + Request + RegExp is also handy.
For me the above two choices are almost equivalent, but the main I JS more familiar, now choose node platform will be more.
On-scale full-site crawl:
Python + scrapy
If the above two programs in the DIY spider is millet plus rifles, that scrapy is simply a heavy cannon, easy to use, custom crawl rules, HTTP error handling, xpath,rpc,pipeline mechanism and so on. Moreover, because Scrapy is based on twisted implementation, so at the same time have a very good efficiency, the only drawback is that the installation is more troublesome, rely on more, I am a relatively new OSX, like no way directly pip install Scrapy
In addition, if you introduce XPath into the spider, and then install the XPath plugin on Chrome, the parsing path is very straightforward and the development efficiency is very high. PHP and JS are not born to do this, Python has a relatively perfect framework, but I have not used it very clearly, but Nodejs can be brought to talk about, because to see
The data is captured with node.
I guess it's like me. Many people are developing and deploying to Linux servers in Windows. Nodejs at this point has a very prominent advantage: ease of deployment, cross-platform almost barrier-free, compared to python ... It's just a peel off.
Parsing the page with Cheerio, fully compatible with jquery syntax, familiar with the front-end words with a very refreshing, no longer do not toss annoying regular;
The operation of the database directly with the MySQL module on the line, the function is all;
Crawl efficiency, actually did not actually do the stress test, because I caught the knowledge, the thread a little more bottleneck ran to the bandwidth. And it is not really multithreaded but asynchronous, the last bandwidth full (about hundreds of threads, 10mb/s around), the CPU is only about 50%, this is only a Linode minimum host CPU. In addition, I usually limit the thread and crawl interval, it is not very expensive performance;
Finally, the code, asynchronous programming the most headache is to fall into the callback hell, according to their actual situation to write a multi-line queue, it is not much more trouble than synchronous programming just. 1.1-point Answer:
1. The ability to parse the page
Regarding this article, basically is to rely on the specific language third party package to complete the webpage parsing. If you want to implement an HTML parser from scratch, the obstacles to difficulty and time are great. For complex web pages or requests that are generated based on a large number of JavaScript operations, you can do so by scheduling the browser environment. On this one, Python is absolutely competent.
2. Ability to operate the database (MySQL)
Python has an official and third-party connection library to the database's operational capabilities. In addition, for crawling data, stored in a NoSQL database is considered more appropriate for the individual.
3. Crawl Efficiency
It is true that scripting languages do not operate at a high speed, but the speed of these languages is negligible relative to the strength of the anti-crawler mechanism of a particular website and the speed of the network IO, but at the level of the developer. If you take advantage of the waiting time for sending network requests to handle other things (multithreading, multi-process, or co-processes), then the efficiency of each language is not a problem.
4. Code Volume
This is a point where Python is a dominant, well known Python code, as long as the developer level, Python code can be as straightforward as pseudo-code, and the code is low.
when recommending a language, please describe the required class library or framework.
For example: Python+mysqldb+urllib2+re
Python:requests + MongoDB + beautifulsoup
PS: Actually I'm not very fond of using python (probably on the Windows platform for reasons that require a variety of transcoding characters, and multithreading seems to be very chicken.) )
Because of the Gil's existence, Python's multithreading does not take advantage of multicore advantages, which you can use to solve multiple processes. However, for the crawler, more time is the network IO waiting, so the direct use of the co-process can be a good way to improve the crawl speed.
In addition, I recently summed up some python writing experience in my column, if interested in welcoming onlookers.
Column Address: / http zhuanlan.zhihu.com/xlz- D Use Python to crawl and extract useful content from HTML, because the library is really handy, beautifulsoup and requests.
Using node to write to the database, the asynchronous way does not have to wait for the completion of synchronous IO, and does not need to involve multi-threaded lock problem. Now that node5.x has supported ES6, you can use promise to solve the problem of multiple nested callback functions.
As for data capture and analysis with PHP, or forget it. I've written a crawl script in PHP node. js Python, so let's talk about it briefly.
First PHP. First of all, the advantage: the Web crawl and parse the framework of the HTML grab a lot, a variety of tools directly to use on the line, more peace of hand. Disadvantage: First speed/efficiency is a problem, one time to download the movie poster, because it is crontab regular execution, also did not optimize, open the PHP process too much, directly to the memory burst. Then the grammar is also very procrastination, a variety of key symbols too much, not concise enough to give people a feeling of not seriously designed, write up very troublesome.
node. js. The advantage is efficiency, efficiency or efficiency, because the network is asynchronous, so basically like hundreds of process concurrency as strong, memory and CPU consumption is very small, if not to fetch the data to perform complex processing, then the system bottleneck is basically in the bandwidth and write database such as MySQL I/O speed. Of course, the pros of the reverse is also a disadvantage, the asynchronous network represents you need to callback, this time if the business needs are linear, such as must wait for the previous page crawl completed, to get the data before the next page crawl, or even multilayer dependencies, it will appear scary multi-layered callback! Basically, the code structure and logic will be mess. Of course, you can use step and other process control tools to solve these problems.
Finally, Python. If you have no extreme requirements for efficiency, then it is recommended to use python! First of all, Python's syntax is very concise, the same statement, you can knock the keyboard many times. Python is then well-suited for data processing, such as package unpacking of function parameters, list parsing, and matrix processing.
I've been working on a Python Data Capture toolkit recently, and it's still in the process of modification, welcome Star:yangjiepro/cutout-github
Python has scapy, a framework designed to be used as a crawler to crawl numbers in the mobile Verification Code platform with Curl in PHP
Use curl to crawl the grass Liu page and automatically download pictures
Well, I like the grass, Python, I'm still watching, I think, Python is really powerful, Nodejs will see later,
Oh, PHP does not support multi-threading, so can only use the server or extension to do, MAB, I will not ...
Forget it, look at the grass ... Python is recommended, and the multithreading aspect can be very cool.
Once used Python to write the crawler of the 8 major music sites, so it is very important to recommend you. I have used PHP, Python,js to write crawlers have not seen, node. JS does not understand.
php Write crawler ok, I wrote a, with PHP Command line under the run. With Curl_multi 50 thread concurrency, a day can catch about 600,000 pages, depending on the speed, I am using the campus network so faster, the data is extracted with the regular.
Curl is a relatively mature lib, exception handling, HTTP header, post and so on are doing very well, the important thing is to operate MySQL under PHP storage operation is more worry.
However, in multi-threaded Curl (Curl_multi), for beginners will be more troublesome, especially the official PHP document in Curl_multi This aspect of the introduction is also very vague.
python write crawler One of the biggest advantage is compared to fool, requests such as the LIB function and curl quite, but if just do simple crawler, ease of use is better, and have
Beautiful Soup Such a fool Lib, It's really good for crawlers.
But coding may indeed be a headache for beginners, I think PHP may be a little better, in fact, if not the team, I do the crawler, I have to write in PHP.
JavaScript I feel like a virtual machine in a virtual machine, throw aside
performance .
- It is first in a sandbox run, for the operation of the database or local files , it will be more troublesome, no native interface, I did not use this as a crawler, nor to study what other programs.
- For Dom Tree parsing, in addition to inefficient, memory consumption is relatively large .
- Cross-domain, though, can be disabled by--disable-web-security under Chrome, but it's also a hassle.
- In short JS to write crawler, trouble is a lot of things.
I've never seen anyone write a reptile with this.
node. js I'm really not used to it.
1. The parsing ability of the page is basically no different, we all support the regular, but python some fools to expand, it will be more convenient to use;
2. The ability to operate the database, PHP has native support for MySQL, Python needs to add mysqldb such as Lib, but also not trouble;
3. Crawl efficiency, all support multi-threading, efficiency I do not feel the difference, basically the bottleneck is only on the network. But rigorous testing I did not do, after all, I do not use a variety of languages to achieve the same function of the habit, but I feel that PHP seems to be faster?
4. The amount of code, crawler This simple thing basically no difference, dozens of rows of things, if you add exception handling also hundred line, or the trouble point of the mark down, and so on the heavy climb, and so on the processing, also hundreds of lines, we have no difference.
But Python is clearly the least if it doesn't count Lib in.
when it comes to performance, reptiles and performance are basically not on the edge, it's a matter of no consideration.。 When I open the crawler, will be nearly 30Mbps crawl efficiency, with PHP Command line to do the crawler, CPU occupied also 3-5%, memory consumption is probably 15-20mib (Core 2 Duo p8700--some historical old u, crawler is 50 threads, Each thread contains 10 regular extracts, 1 JSON parsing, 2 database insert operations (if not EXIST of millions data), and 40 or so of various anomaly judgments)--
bottlenecks should only be network。
In the case of you do not have g mouth, do not control what performance, casually pick a all the same, oneself familiar with good.
I drove the crawler those days, probably climbed about 270GiB of data. I drove the crawler those days, probably climbed about 270GiB of data.
