PHP CURL Crawl web post data and other

Source: Internet
Author: User
Tags curl php tutorial urlencode

Using PHP's Curl library can be a simple and effective way to capture web pages. All you have to do is run a script and then analyze the pages you crawl, and then you can get the data you want in a program. Whether you want to take part of the data from a link, or take an XML file and import it into a database tutorial, it's a simple way to get web content, CURL is a powerful PHP library. This article mainly describes if you use this PHP library.

Enable CURL settings

First, we have to determine if our PHP is open to this library, and you can get this information by using the PHP tutorial _info () function.

<?php
Phpinfo ();
?>

If you can see the following output on a Web page, then the Curl Library has been opened.

If you see it then you need to set up your PHP and open the library. If you are under the Windows platform, then it is very simple, you need to change your php.ini file settings, find Php_curl.dll, and cancel the previous semicolon comment on the line. As shown below:

Cancel the comment below
Extension=php_curl.dll

If you are under Linux, then you need to recompile your PHP, edit, you need to open the compilation parameters--The Configure command on the "–with-curl" parameter.

A small example

If everything is ready, here is a small routine:

<?php
Initializes a CURL object
$curl = Curl_init ();

Set the URL you want to crawl
curl_setopt ($curl, Curlopt_url, ' http://cocre.com ');

Set Header
curl_setopt ($curl, Curlopt_header, 1);

Sets the curl parameter to require that the results be saved to the string or to the screen.
curl_setopt ($curl, Curlopt_returntransfer, 1);

Run Curl, request Web page
$data = curl_exec ($curl);

Close URL Request
Curl_close ($curl);

Show the data you get
Var_dump ($data);
?>

How to post data

This is the code that crawls the page, and the following is the post data to a Web page. Let's say we have a URL http://www.example.com/sendSMS.php that handles the form, it can accept two form fields, one is the phone number, the other is a text message.

<?php
$phoneNumber = ' 13912345678 ';
$message = ' This is generated by curl and PHP ';
$curlPost = ' pnumber= '. UrlEncode ($phoneNumber). ' &message= '. UrlEncode ($message). ' &submit=send ';
$ch = Curl_init ();
curl_setopt ($ch, Curlopt_url, ' http://www.example.com/sendSMS.php ');
curl_setopt ($ch, Curlopt_header, 1);
curl_setopt ($ch, Curlopt_returntransfer, 1);
curl_setopt ($ch, Curlopt_post, 1);
curl_setopt ($ch, Curlopt_postfields, $curlPost);
$data = Curl_exec (); Curl_close ($ch);
?>

From the above program we can see that we set the Post method of the HTTP protocol using Curlopt_post instead of the Get method, and then set the post data in Curlopt_postfields.

About proxy servers

The following is an example of how to use a proxy server. Note that the highlighted code, the code is very simple, I do not have to say more.

<?php
$ch = Curl_init ();
curl_setopt ($ch, Curlopt_url, ' http://www.example.com ');
curl_setopt ($ch, Curlopt_header, 1);
curl_setopt ($ch, Curlopt_returntransfer, 1);
curl_setopt ($ch, Curlopt_httpproxytunnel, 1);
curl_setopt ($ch, Curlopt_proxy, ' fakeproxy.com:1080 ');
curl_setopt ($ch, curlopt_proxyuserpwd, ' User:password ');
$data = Curl_exec (); Curl_close ($ch);
?>

About SSL and cookies

With regard to SSL, the HTTPS protocol, you only need to turn the http://in the Curlopt_url connection into an HTTPS tutorial://It's OK. Of course, there is also a parameter called Curlopt_ssl_verifyhost can be set to verify the site.

For cookies, you need to know the following three parameters:

Curlopt_cookie, set a COOKIE in your face-to-face session

Curlopt_cookiejar, save a cookie when the session ends

Curlopt_cookiefile,cookie's file.

HTTP Server Authentication

Finally, let's look at HTTP server authentication.

<?php
$ch = Curl_init ();
curl_setopt ($ch, Curlopt_url, ' http://www.example.com ');
curl_setopt ($ch, Curlopt_returntransfer, 1);
curl_setopt ($ch, Curlopt_httpauth, Curlauth_basic);
curl_setopt (Curlopt_userpwd, ' [Username]:[password] ')

$data = Curl_exec ();
Curl_close ($ch);
?>

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.