In fact, the thief program uses a specific function in php to collect the content of other websites, and then saves the content we want to save to our local database through regular expression analysis, next I will introduce the implementation method of the php thief program. If you need it, please refer to it.
In the following data collection process, the file_get_contents function is critical. Let's take a look at the syntax of the file_get_contents function.
String file_get_contents (string $ filename [, bool $ use_include_path = false [, resource $ context [, int $ offset =-1 [, int $ maxlen])
Like file (), only file_get_contents () can read the file into a string. The content with the length of maxlen will be read at the position specified by the offset parameter. If it fails, file_get_contents () returns FALSE.
The file_get_contents () function is used to read the file content into a string. If the operating system supports it, the memory ing technology will be used to enhance the performance.
Example
| The Code is as follows: |
Copy code |
|
<? Php
$ Homepage = file_get_contents ('HTTP: // www.hzhuti.com /');
Echo $ homepage;
?> |
In this way, $ homepage is the content of the collected network, which is saved. Let's get started with so much.
Example
| The Code is as follows: |
Copy code |
|
<? Php
Function fetch_urlpage_contents ($ url ){
$ C = file_get_contents ($ url );
Return $ c;
}
// Obtain the Matching content
Function fetch_match_contents ($ begin, $ end, $ c)
{
$ Begin = change_match_string ($ begin );
$ End = change_match_string ($ end );
$ P = "{$ begin} (. *) {$ end }";
If (eregi ($ p, $ c, $ rs ))
{
Return $ rs [1];}
Else {return "";}
} // Escape a regular expression string
Function change_match_string ($ str ){
// Note: The following is a simple escape
// $ Old = array ("/", "$ ");
// $ New = array ("/", "$ ");
$ Str = str_replace ($ old, $ new, $ str );
Return $ str;
}
// Collect web pages
Function pick ($ url, $ ft, $ th)
{
$ C = fetch_urlpage_contents ($ url );
Foreach ($ ft as $ key => $ value)
{
$ Rs [$ key] = fetch_match_contents ($ value ["begin"], $ value ["end"], $ c );
If (is_array ($ th [$ key])
{Foreach ($ th [$ key] as $ old => $ new)
{
$ Rs [$ key] = str_replace ($ old, $ new, $ rs [$ key]);
}
}
}
Return $ rs;
}
$ Url = "http://www.bkjia.com"; // address to be collected
$ Ft ["title"] ["begin"] = "<title>"; // start point of Interception
$ Ft ["title"] ["end"] = "</title>"; // the end point of the screenshot.
$ Th ["title"] ["Zhongshan"] = "Guangdong"; // Replace the intercepted part
$ Ft ["body"] ["begin"] = "<body>"; // start point of Interception
$ Ft ["body"] ["end"] = "</body>"; // the end point of the screenshot.
$ Th ["body"] ["Zhongshan"] = "Guangdong"; // Replace the intercepted part
$ Rs = pick ($ url, $ ft, $ th); // start collection
Echo $ rs ["title"];
Echo $ rs ["body"]; // output
?> |
The following code is used to extract all hyperlinks, mailboxes, or other specific content from the web page.
| The Code is as follows: |
Copy code |
|
<? Php
Function fetch_urlpage_contents ($ url ){
$ C = file_get_contents ($ url );
Return $ c;
}
// Obtain the Matching content
Function fetch_match_contents ($ begin, $ end, $ c)
{
$ Begin = change_match_string ($ begin );
$ End = change_match_string ($ end );
$ P = "# {$ begin} (. *) {$ end} # iU"; // I indicates case-insensitive. U prohibits greedy matching.
If (preg_match_all ($ p, $ c, $ rs ))
{
Return $ rs ;}
Else {return "";}
} // Escape a regular expression string
Function change_match_string ($ str ){
// Note: The following is a simple escape
$ Old = array ("/", "$ ",'? ');
$ New = array ("/", "$ ",'? ');
$ Str = str_replace ($ old, $ new, $ str );
Return $ str;
}
// Collect web pages
Function pick ($ url, $ ft, $ th)
{
$ C = fetch_urlpage_contents ($ url );
Foreach ($ ft as $ key => $ value)
{
$ Rs [$ key] = fetch_match_contents ($ value ["begin"], $ value ["end"], $ c );
If (is_array ($ th [$ key])
{Foreach ($ th [$ key] as $ old => $ new)
{
$ Rs [$ key] = str_replace ($ old, $ new, $ rs [$ key]);
}
}
}
Return $ rs;
}
$ Url = "http://www.bkjia.com"; // address to be collected
$ Ft ["a"] ["begin"] = '<a'; // the start point of the screenshot. <br/>
$ Ft ["a"] ["end"] = '>'; // the end point of the screenshot.
$ Rs = pick ($ url, $ ft, $ th); // start collection
Print_r ($ rs ["a"]);
?> |
TipsFile_get_contents is easy to prevent from being collected. We can use curl to simulate user access to the website, which is much more advanced than above. file_get_contents () is less efficient, common failures and high curl () efficiency. multithreading is supported, but curl extension must be enabled. The steps to enable curl extension are as follows:
1. Copy the three php_curl.dll, libeay32.dll, and ssleay32.dll files in the PHP folder to system32;
2. Remove the semicolon from php. ini (c: WINDOWS Directory) and extension = php_curl.dll;
3. Restart apache or IIS.
A simple page capture function with the Referer and User_Agent spoofing Function
| The Code is as follows: |
Copy code |
|
<? Php
Function GetSources ($ Url, $ User_Agent = '', $ Referer_Url ='') // capture a specified page
{
// $ Url: the Url of the page to be crawled
// $ User_Agent the user_agent information to be returned, such as "baiduspider" or "googlebot"
$ Ch = curl_init ();
Curl_setopt ($ ch, CURLOPT_URL, $ Url );
Curl_setopt ($ ch, CURLOPT_USERAGENT, $ User_Agent );
Curl_setopt ($ ch, CURLOPT_REFERER, $ Referer_Url );
Curl_setopt ($ ch, CURLOPT_FOLLOWLOCATION, 1 );
Curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1 );
$ MySources = curl_exec ($ ch );
Curl_close ($ ch );
Return $ MySources;
}
$ Url = "http://www.bkjia.com"; // NO content to get
$ User_Agent = "baiduspider + (+ http://www.baidu.com/search/spider.htm )";
$ Referer_Url = 'HTTP: // www.jb51.net /';
Echo GetSources ($ Url, $ User_Agent, $ Referer_Url );
?> |