Various solutions for Web data scraping

For Internet people, web data scraping has become an urgent and real requirement. In today's open source era, the problem is often not whether there is a solution, but how to choose the right solution for you, because there are always a lot of potential options for you to choose from. Web data scraping of course is no exception, I think, we have at least four types of programs can choose,

1. Web scraping Software

Dedicated data acquisition software does not require you to write any code, you only need to configure some information, such as the target URL of the acquisition, interest fields and so on. However, learning how to use these software is a very laborious task, most of which require a certain technical background, and often need to know the following techniques: XPath, HTML, regular expression. The author lists several of this software as below,

• Locomotive collector----Support output to various databases
• A1 Website Scraper------Support output to CSV file;
• bget Web Data Extraction-----Support output to database and CSV file;

For more information on this software, refer to this document: collections of Web scraping software and server

2. Web scraping framework

The scraping framework is probably the best choice for developer because it is powerful and efficient, and has a framework for different platforms to choose from, such as:

• Scrapy, Python Platform
• Upton, Ruby Platform
• Pismo, Ruby Platform
• Nokogiri, Ruby Platform
• Node.io, node. JS Platform
• Crawler, node. JS Platform

The author believes that Python has a strong advantage in data processing, more importantly, Python is also very suitable for the development of real products, the author is persistent and strongly recommend Scrapy as the first choice of web scraping.

3. Web Scraping Service

If you're just trying to get data and don't want to take the hassle out of learning specific scraping technologies, a reliable web scraping service provider may be your best choice, with a few of the more well-known web scraping services:

• Http://80legs.com/services.html
• http://www.grepsr.com/
• http://scrapinghub.com/
• http://www.connotate.com/
• http://scrapinghub.com/

4. Other options

Selenium was originally a browser automation Test tool, but it can also be used for web scraping. If the content you are collecting is automatically generated by JavaScript, you may need to selenium because the HTML parser in other scraping scenarios often do not support JavaScript, such as Scrapy. In addition, there are many sites prohibit scraping, this time you also need to selenium, because selenium is completely simulated people's browsing behavior, selenium is essentially working on the browser.

Of course, we can also choose a lot of JavaScript-based scraping scenarios, such as phantomjs  is a very good choice, PHANTOMJS essentially simulates the browser.