With PHP's curl library you can simply and efficiently grab a webpage, you just run a script, then analyze the page you crawled, and then you can get the data you want in a program. Whether you want to take part of the data from a link, or take an XML file and import it into the database, curl is a powerful PHP library, even if you simply get the Web content. This article mainly describes if you use this PHP library.
Enable Curl Settings
First, we have to make sure that our PHP opens the library, and you can get this information by using the Php_info () function:
If you can see the following output on a webpage, it means that the Curl Library is turned on.
If you see it, then you need to set up your PHP and open the library. If you are under the Windows platform, then very simple, you need to change the settings of your php.ini file, find Php_curl.dll, and cancel the previous semicolon comment on the line, as follows:
Cancel the comment under the Extension=php_curl.dll |
If you are under Linux, then you need to recompile your PHP, edit, you need to open the compilation parameters, the Configure command to add the "–with-curl" parameter.
A small example
If everything is ready, here's a small example:
Initialize a Curl Object $curl = Curl_init (); Set the URL you need to crawl curl_setopt ($curl, Curlopt_url, ' http://www.example.com '); Set Header curl_setopt ($curl, Curlopt_header, 1); Set the curl parameter to require the result to be saved to a string or output to the screen curl_setopt ($curl, Curlopt_returntransfer, 1); Run Curl, request a Web page $data = curl_exec ($curl); Close URL Request Curl_close ($curl); Show the data obtained Var_dump ($data); ?> |
How to post data
The above is the code to crawl the Web page, the following is the post data to a page. Suppose we have a URL http://www.example.com/sendSMS.php that handles the form, which can accept two form fields, one is the phone number, and the other is the message content. The sample code is as follows:
$phoneNumber = ' 13912345678′; $message = ' This message is generated by curl and PHP '; $curlPost = ' pnumber= '. UrlEncode ($phoneNumber). ' &message= '. UrlEncode ($message). ' &submit=send '; $ch = Curl_init (); curl_setopt ($ch, Curlopt_url, ' http://www.example.com/sendSMS.php '); curl_setopt ($ch, Curlopt_header, 1); curl_setopt ($ch, Curlopt_returntransfer, 1); curl_setopt ($ch, Curlopt_post, 1); curl_setopt ($ch, Curlopt_postfields, $curlPost); $data = Curl_exec (); Curl_close ($ch); ?> |
From the above program we can see that using Curlopt_post to set the HTTP protocol post method instead of the Get method, and then set the post data to Curlopt_postfields.
About proxy servers
Here is an example of how to use a proxy server, the code is very simple, I will not have to say more:
$ch = Curl_init (); curl_setopt ($ch, Curlopt_url, ' http://www.example.com '); curl_setopt ($ch, Curlopt_header, 1); curl_setopt ($ch, Curlopt_returntransfer, 1); curl_setopt ($ch, Curlopt_httpproxytunnel, 1); curl_setopt ($ch, Curlopt_proxy, ' fakeproxy.com:1080′); curl_setopt ($ch, curlopt_proxyuserpwd, ' User:password '); $data = Curl_exec (); Curl_close ($ch); ?> |
About SSL and cookies
With respect to SSL, the HTTPS protocol, you only need to turn the http://To https://on the Curlopt_url connection. Of course, there is also a parameter called Curlopt_ssl_verifyhost that can be set to verify the site.
For cookies, you need to know the following three parameters:
1, Curlopt_cookie: Set a COOKIE in the face of the conversation
2. Curlopt_cookiejar: Save a cookie when the session ends
3, Curlopt_cookiefile:cookie files.
HTTP Server Authentication
Finally, let's look at the HTTP Server Authentication scenario:
$ch = Curl_init (); curl_setopt ($ch, Curlopt_url, ' http://www.example.com '); curl_setopt ($ch, Curlopt_returntransfer, 1); curl_setopt ($ch, Curlopt_httpauth, Curlauth_basic); curl_setopt (Curlopt_userpwd, ' [Username]:[password] ') $data = Curl_exec (); Curl_close ($ch); ?> |