PHP implementation of the Thief program Instance _php example

Why use the "Thief program"?

Remote Crawl article information or merchandise is a lot of enterprises require programmers to achieve the function, that is, Vulgar said the thief program. Its main advantages are: The solution of the company's network of heavy work, greatly improve the efficiency. Only need to run quickly to crawl the information of other people's websites.

Where does the "Thief program" run?

"Thief program" should be under Windows dos or Linux under the PHP command to run the best, because the Web page will run overtime.

For example, in the case of Windows DOS:

The realization of "thief procedure"

Here mainly through an example to explain, we grasp the "Huaqiang Electronic Network" information, please look at this link http://www.hqew.com/info-c10.html, when you open this page will find some phenomena:

1, the information list has 500 pages (2012-01-03);

2, each page of the URL links have laws, such as: The 1th page for http://www.hqew.com/info-c10-1.html, 2nd page for http://www.hqew.com/info-c10-2.html; The No. 500 page is http://www.hqew.com/info-c10-500.html;

3, from the 2nd can be known, "Huaqiang Electronic Network" information is pseudo static or generated static page

In fact, most of the site has such a law, such as: Zhongguancun, HC NET, Sina, Taobao ...

In this way, we can use this idea to achieve the content of the page crawl:
1. Get the content of the article List page first;
2, according to the article List page content cycle to get the URL address of the article;
3, according to the article URL to get the details of the article

Here, we mainly crawl the information page: title, release as expected (date), author (author), source (source), contents (content)

"Huaqiang Electronic Network" information crawl

First, you build the data table structure, as follows:

CREATE TABLE ' article '. ' article ' ( 
' id ' mediumint (8) UNSIGNED not NULL auto_increment KEY, 
' title ' PRIMARY  HAR (255) CHARACTER SET UTF8 COLLATE utf8_general_ci not null, 
' date ' VARCHAR is not null, 
' author ' VARCHAR ( CHARACTER SET UTF8 COLLATE utf8_general_ci not NULL, 
' source ' VARCHAR (MB) CHARACTER SET UTF8 COLLATE Eneral_ci NOT NULL, 
' content ' TEXT is not null 

Crawl program:

<?php/** * Grasping "Huaqiang Electronic Network" information program * Author Lee. * Last modify $Date: 2012-1-3 15:39:35 $ */header (' content-type:text/html; 
Charset=utf-8 '); $mysqli = new mysqli (' localhost ', ' root ', ' 1715544 ', ' article '); # database connection, please manually modify your own database information $mysqli->set_charset (' UTF8 '); 
  # Set database encoding function data ($url) {global $mysqli; $result = file_get_contents ($url); # $result Get URL link content (Note: Here is the link to the article list) $pattern = '/<li><span class= ' box_r ' >.+<\/span><a href= ' ([^]]+ ) "title=". + ">.+<\/a><\/li>/usi"; # Get the URL of the article matching regular preg_match_all ($pattern, $result, $arr); # Assign the article list URL to the array $arr (two-dimensional array) foreach ($arr [1] as $val) {$val = ' http://www.hqew.com '. $val; # Real article URL address $r E = file_get_contents ($val); # $re for the content of the article URL $pa = '/<div id= ' article ' >\s+ 

Through the above procedures, you can achieve the grasp of Huaqiang electronic network information.

Entry Method Init ($min, $max) If you want to crawl 1-500 page content, then init (1, 500) can! In this way, in less than a long time, Huaqiang electronic network of information will be all crawled into the database inside. ^_^

Executive interface:

Database: