PHP Crawler's million-level knowledge of user data crawl and analysis _php instance

This time grabbed 1.1 million of the user data, the data analysis results are as follows:

Pre-development preparation

• Install the Linux system (Ubuntu14.04) and install an Ubuntu under the VMware virtual machine;
• Install PHP5.6 or above version;
• Install MySQL5.5 or above version;
• Install curl, pcntl extensions.

Using PHP's Curl extension to crawl page data

PHP's Curl extension is a library of PHP-supported libraries that allow you to connect and communicate with a variety of servers using various types of protocols.

This program is to crawl the user data, to be able to access the user's personal pages, users need to log in to access. When we click on a user avatar link in the browser's page to enter the user's Personal center page, the reason why you can see the user's information is that when you click on the link, the browser helps you to bring the local cookies to the new page, so you can access the user's Personal center page. So before you can access a personal page, you need to obtain the user's cookie information, and then bring the cookie information every time you curl the request. In terms of getting cookie information, I used my own cookie to see my cookie information on the page:

Copy each, to "__utma="? __utmb=?; " This form consists of a cookie string. The cookie string can then be used to send the request.

The initial example:

$url = ' http://www.zhihu.com/people/mora-hu/about '; Here Mora-hu represents the user id
$ch = curl_init ($url);//Initialize session
curl_setopt ($ch, Curlopt_header, 0);
curl_setopt ($ch, Curlopt_cookie, $this->config_arr[' User_cookie ')); Set Request Cookie
curl_setopt ($ch, curlopt_useragent, $_server[' http_user_agent ');
curl_setopt ($ch, Curlopt_returntransfer, 1); The information obtained by CURL_EXEC () is returned as a file stream, rather than as a direct output.
curl_setopt ($ch, curlopt_followlocation, 1); 
$result = curl_exec ($ch);
return $result; The results of the crawl

Run the above code to get the Mora-hu user's Personal center page. Using this result and using regular expressions to process the page, we can get the information that the name, gender and so on need to crawl.

1, picture anti-theft chain

When outputting personal information after regular processing of the returned result, it is found that the user's avatar cannot be opened on the page. After access to information that is because of the picture did the anti-theft chain treatment. The solution is to request a picture to forge a referer in the request.

After you use a regular expression to get a link to a picture, send another request, and then bring the source of the picture request, indicating that the request came from a known site forwarding. Specific examples are as follows:

function getimg ($url, $u _id)
{
  if file_exists ('./images/'. $u _id. ". jpg")
  {return
    "images/$u _id". '. jpg ';
  }
  if (empty ($url))
  {return
    ';
  }
  $context _options = Array ( 
    ' http ' => 
    array (
      ' header ' => ' referer:http://www.zhihu.com '/ With the Referer parameter
  )
);

  $context = Stream_context_create ($context _options); 
  $img = file_get_contents (' http: '. $url, FALSE, $context);
  File_put_contents ('./images/'. $u _id. ". jpg", $img);
  Return "images/$u _id". '. jpg ';
}

2, crawling more users

Once you have crawled your personal information, you will need to access the user's followers and the list of interested users to get more user information. And then one level at a a-tier access. As you can see, in the Personal center page, there are two links as follows:

Here are two links, one is attention, the other is the concern, with the "Attention" link as an example. Use a regular match to match to the corresponding link, get the URL and then use the curl with cookies to send a request again. After grabbing the user's attention for a list page, you can get the following page:

Analyze the HTML structure of the page, because as long as you get the user's information, so just need to frame this piece of div content, the user name is here. As you can see, the URL for the user's attention page is:

The URL for different users is almost the same, and the difference is in the username. Use a regular match to get the list of user names, one by one to spell the URL, and then send a request (of course, one is slower, the following is a solution, this will be said later). After entering the new user's page, repeat the steps above, and cycle until you reach the amount of data you want.

3. Number of Linux statistics files

After running for a while, the script needs to see how many pictures it gets, and when the volume of data is large, it's a bit slow to open the folder to see the pictures. Scripts are run in a Linux environment, so you can use Linux commands to count the number of files:

Ls-l | grep "^-" | Wc-l

Where Ls-l is the long list output the file information in the directory (the files here can be directories, links, device files, and so on); grep "^-" filters long list output information, "^-" retains only generic files, if only the directory is "^d"; Wc-l is the number of rows for the statistic output information. Here is a running example:

4. Duplicate data processing when inserting MySQL

After the program has been running for some time, it is found that many of the user's data are duplicated, and therefore need to be processed when inserting duplicate user data. The processing programme is as follows:

1 Check whether the data already exists in the database before inserting the database;

2 to add a unique index, insert with inserts into ... On DUPLICATE KEY UPDATE ...

3) Add a unique index, insert using the inserts Ingnore into ...

4) Add a unique index, insert with REPLACE into ...

The first scenario is the simplest but also the least efficient, and is therefore not taken. The results of the two and four scenarios are the same, and the difference is that when the same data is encountered, INSERT into ... On DUPLICATE KEY Update is directly updated, and REPLACE into deletes the old data and inserts the new one, and in the process, the index needs to be maintained again, so it's slow. So the second option was chosen between the two and 42 of them. In the third scenario, insert Ingnore ignores errors that occur when an INSERT statement is executed, does not ignore syntax problems, but ignores the presence of primary keys. This makes it much better to use an INSERT ingnore. Finally, considering the number of bars to record duplicate data in the database, the second scenario is used in the program.

5, using Curl_multi to achieve multi-threaded crawl page

Just start a single process and a single curl to crawl data, speed is very slow, hanging machine climbed a night can only catch 2W of data, and then think of can enter the new user page Curl request a one-time request multiple users, then found Curl_multi this good thing. Curl_multi Such functions can request multiple URLs at the same time, rather than a single request, which is similar to the ability of a process in a Linux system to run multiple threads. Here is an example of using Curl_multi to implement a multithreaded crawler:

  $MH = Curl_multi_init (); Returns a new curl batch handle for ($i = 0; $i < $max _size $i + +) {$ch = Curl_init ();//initialization of a single Curl session curl_setopt ($ch, CUR
    Lopt_header, 0); curl_setopt ($ch, Curlopt_url, ' http://www.zhihu.com/people/'. $user _list[$i].
    '/about ');
    curl_setopt ($ch, Curlopt_cookie, self:: $user _cookie); curl_setopt ($ch, Curlopt_useragent, ' mozilla/5.0 (Windows NT 6.1;
    WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/44.0.2403.130 safari/537.36 ');
    curl_setopt ($ch, Curlopt_returntransfer, true);
    curl_setopt ($ch, curlopt_followlocation, 1);
    $requestMap [$i] = $ch; Curl_multi_add_handle ($MH, $ch);
  Add a separate curl handle} to the Curl batch session $user _arr = Array ();

    do {///Run the child connection of the current CURL handle while (($cme = Curl_multi_exec ($MH, $active)) = = Curlm_call_multi_perform);
          if ($cme!= curlm_ok) {break;}
      Gets the associated transport information for the currently parsed curl while ($done = Curl_multi_info_read ($MH)) {$info = Curl_getinfo ($done [' handle ']); $tmp _result = Curl_mulTi_getcontent ($done [' handle ']);

      $error = Curl_error ($done [' handle ']);

      $user _arr[] = array_values (GetUserInfo ($tmp _result)); Ensure that there are simultaneous $max_size requests in the process if ($i < sizeof ($user _list) && isset ($user _list[$i]) && $i < count ($use
        R_list)) {$ch = Curl_init ();
        curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_url, ' http://www.zhihu.com/people/'. $user _list[$i].
        '/about ');
        curl_setopt ($ch, Curlopt_cookie, self:: $user _cookie); curl_setopt ($ch, Curlopt_useragent, ' mozilla/5.0 (Windows NT 6.1;
        WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/44.0.2403.130 safari/537.36 ');
        curl_setopt ($ch, Curlopt_returntransfer, true);
        curl_setopt ($ch, curlopt_followlocation, 1);
        $requestMap [$i] = $ch;

        Curl_multi_add_handle ($MH, $ch);
      $i + +;
    } curl_multi_remove_handle ($MH, $done [' handle ']);
  if ($active) curl_multi_select ($MH, 10); } while ($active);
  Curl_multi_close ($MH);

 return $user _arr;

6, HTTP 429 Too Many Requests

With the Curl_multi function, you can send multiple requests at the same time, but when you make 200 requests at the same time, you find that many requests cannot be returned, that is, the case where the packet was dropped. Further analysis, using the Curl_getinfo function to print each request handle information, returns an associative array containing HTTP response information, one of which is Http_code, which indicates the HTTP status code that the request returns. See a lot of requests http_code are 429, this return code means to send too many requests. I guess it was done with the protection of the crawler, so I took other sites to do the test, found that a one-time issue of 200 requests no problem, proved my guess, know that in this aspect of the protection, that the number of one-time requests is limited. So I kept reducing the number of requests and found that there were no packets lost at 5. Note In this program can only send 5 requests a time, although not many, but this is a small upgrade.

7, use Redis to save the users who have already visited

In the process of crawling the user, it is found that some users have been visited, and his followers and interested users have been acquired, although the database at the level of duplication of data processing, but the program will still use curl to send a request, so repeated sending requests have a lot of duplication of network overhead. Another is that the user to be crawled needs to be temporarily saved in one place for the next execution, at first it was put into the array, and later found to add multiple processes to the program, in the process of programming, the child will share the program code, function library, but the process uses the variable and the other processes used by different. Variables between different processes are separate and cannot be read by other processes, so arrays cannot be used. So the idea is to use the Redis cache to save the processed users and the users to crawl. This will push the user into a already_request_queue queue each time it is executed, push the user to be crawled (that is, a list of the users ' followers and attention) to the request_queue, and then each time it is executed from the request _queue Pop a user, and then judge whether in the Already_request_queue, if in, then proceed to the next, or continue execution.

Use the Redis example in PHP:

<?php
  $redis = new Redis ();
  $redis->connect (' 127.0.0.1 ', ' 6379 ');
  $redis->set (' tmp ', ' value ');
  if ($redis->exists (' tmp '))
  {
    echo $redis->get (' tmp '). "\ n";
  }

8, using PHP pcntl extension to achieve multiple processes

Using the Curl_multi function to achieve multi-threaded crawl user information, the program runs one night, the final data has 10W. Can not achieve their own ideal goal, and then continue to optimize, and later found that PHP has a pcntl extension can achieve multiple process programming. The following are examples of multiple programming programming:

PHP Multi-process Demo
//fork10 process for
($i = 0; $i < $i + +) {
  $pid = Pcntl_fork ();
  if ($pid = = 1) {
    echo "could not fork!\n";
    Exit (1);
  }
  if (! $pid) {
    echo "child process $i running\n";
    Quits after the child process has finished executing, lest it continue to fork out the new subprocess
    exit ($i);
  }

Wait for the child process to complete, avoid zombie while
(pcntl_waitpid (0, $status)!=-1) {
  $status = Pcntl_wexitstatus ($status);
  echo "Child $status completed\n";
}

9. View the CPU information of the system under Linux

After the implementation of the process of programming, think of more than a few processes to continue to crawl the user's data, and then opened 8 of the process ran a night after the discovery can only get 20W of data, not much ascension. So lookup data found that, according to the system optimized CPU performance tuning, the program's maximum number of processes can not be casually given, according to the number of CPUs and to give, the maximum number of processes is the best CPU twice times the number of cores. So you need to look at the CPU's information to see the CPU's kernel count. commands for viewing CPU information under Linux:

Cat/proc/cpuinfo

The results are as follows:

Where model name represents the CPU type information, and the CPU cores represents the CPU kernel number. The number of cores here is 1, because it is running under a virtual machine, the number of CPUs allocated is small, so only 2 processes can be opened. The end result was that 1.1 million of the user data was crawled over a weekend.

10. Redis and MySQL connection problems in multi-process programming

Under the multiple process conditions, the program runs for a period of time, found that the data can not be inserted into the database, will report the MySQL too many connections error, Redis is the case.

The following code fails to execute:

<?php for
   ($i = 0; $i < $i + +) {
     $pid = Pcntl_fork ();
     if ($pid = = 1) {
        echo "could not fork!\n";
        Exit (1);
     }
     if (! $pid) {
        $redis = predis::getinstance ();
        Do something   
        exit;
     }
   

The root cause is that when each child process is created, a copy of the parent process is inherited. objects can be copied, but the created connections cannot be copied into multiple, resulting in the fact that each process uses the same Redis connection, each doing its own thing, resulting in a baffling conflict.

WORKAROUND: The > program does not fully guarantee that the parent process will not create a Redis connection instance before the fork process. Therefore, to solve this problem can only depend on the child process itself. Imagine that the problem does not exist if the instance obtained in the subprocess is only relevant to the current process. So the solution is to tweak the static mode of the Redis class instantiation to bind to the current process ID.

The modified code is as follows:

<?php public
   static function getinstance () {
     static $instances = Array ();
     $key = Getmypid ()///get current Process ID
     if ($empty ($instances [$key])) {
        $inctances [$key] = new self ();
     }

     return $instances [$key];
   }

11, PHP Statistics script execution time

Because you want to know how much time each process takes, write a function to count the execution time of the script:

function Microtime_float ()
{
   list ($u _sec, $sec) = Explode (", microtime ());
   Return (Floatval ($u _sec) + floatval ($sec));
}

$start _time = Microtime_float ();

Do something
usleep (m);

$end _time = Microtime_float ();
$total _time = $end _time-$start _time;

$time _cost = sprintf ("%.10f", $total _time);

echo "program cost Total". $time _cost. "S\n";

The above is the entire content of this article, for your reference, I hope to help you learn.