Several methods used to write collection programs using PHP and common problems encountered

This document provides several methods used to write collection programs using PHP and common problems encountered. It is free to download. Abstract: This document is suitable for PHP beginners to learn how to understand and apply web page collection. Several methods used to write collection programs using PHP and common problems encountered: 1. file_get_contents ($ url); Use file_get_contents for collection This document is suitable for PHP beginners to understand and apply web page collection.

Several methods used to write a collection program using PHP and common problems encountered:

1. file_get_contents ($ url );

The following example shows how to use file_get_contents to collect data:

$ Url = "https://www.php1.cn ";

$ Con = file_get_contents ($ url); // The collected content is stored in $ con.


2. Use curl for collection

Using curl to collect data is more flexible than file_get_contents. Currently, many web pages have anti-collection measures. If you use file_get_contents to collect data directly, it will fail.

However, curl can be used to simulate browser information for collection.

The following are examples of curl collection. curl_setopt is a frequently used setting. Please select as needed.

$ Url = "https://www.php1.cn ";

$ Useragent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 )";

$ Header = array ('Accept-Language: zh-cn', 'Connection: Keep-Alive ', 'cache-Control: no-cache ');

$ Ch = curl_init (); // initialize curl

Curl_setopt ($ ch, CURLOPT_REFERER, $ url );

Curl_setopt ($ ch, CURLOPT_HTTPHEADER, $ header); // simulate browser header information

Curl_setopt ($ ch, CURLOPT_USERAGENT, $ useragent); // simulate browser Information

Curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, true); // whether to save the collected content

Curl_setopt ($ ch, CURLOPT_TIMEOUT, 60); // The maximum time allowed for curl execution, in seconds

Curl_setopt ($ ch, CURLOPT_URL, $ url); // the url to be collected

Curl_setopt ($ ch, CURLOPT_HEADER, 1); // whether to save the header information

$ Con = curl_exec ($ ch); // The collected content is stored in $ con.


3. Use snoopy, which is very popular on the Internet, for collection. This is a very powerful collection plug-in, and it is very convenient to use. You can also set an agent in it to simulate browser information.

Instance used:

Require ('snoopy. class. php'); // introduce the Snoopy class file

$ Snoopy = new Snoopy; // initialize the snoopy class

$ Url = "https://www.php1.cn ";

$ Snoopy-> fetch ($ url); // starts to collect content

$ Con = $ snoopy-> results; // Save the collected content to $ con

Note: Set the agent to line 1 of the Snoopy. class. php file. Search for "var $ agent" in this file (in quotation marks ). You can use PHP to obtain the browser content,

Use echo $ _ SERVER ['HTTP _ USER_AGENT ']; To obtain the browser information, copy the echo content to the agent.


Above

The three methods are commonly used for collection using PHP. I personally think that using snoopy should be the simplest. Of course, you may need to use snoopy based on the actual situation.

In combination, these examples are just the most basic collection code. When using them, you need to add corresponding regular expressions as needed to extract the required content.

I encountered the following problems when working on the collection program:

1. The 30-second timeout of the PHP program. The solution is to add set_time_limit (0) before the code is collected. 0 indicates no limit.

2. When collecting data, the returned result is 403. In this case, you need to check whether your code simulates browser information.