"Go" using PHP to create a basic crawler program

Web Crawler, also known as Scrapers, is a web crawler that automatically searches the Internet and extracts what you want from it. The development of the Internet is inseparable from them. Crawlers are the core of search engines, and the smart algorithms find pages that match the keywords you enter.

Google Web crawler will enter your domain name, and then scan all pages of your site, to extract the page title, description, keywords and links-and then return these comments to Google HQ, the content stored in a large database,

Today, I'm glad to tell you how to be your own crawler--it doesn't search the entire internet, it just gets all the links and information for a given URL. [Translate: I changed the example slightly according to the application to get the TV listings].

Generally, you should make sure that you are authorized before the crawler, because it is really a gray area. As I said, the Internet is inseparable from these reptiles, and it is important to understand how they work and how they are created.

To make the implementation easier, we use the most popular network programming language,--php. Don't be afraid you don't understand php--I'll show you how to do each step and explain what it's for. I assume you already know HTML and know how to add links and pictures to HTML documents.

You need to have a server running PHP scripts. It is recommended that you use Appserv.

Entry

We use a secondary library simple HTML DOM. This library is used to easily traverse HTML documents.

First, we test whether the environment is normal. Create a. php file. and copy the following code to the server www folder.

<?php    include_once(‘simple_html_dom.php‘);    phpinfo();?>

Browse through the files created by the browser. If everything is correct, then you will see the server information such as output:

The 1th line of code <?php tells the server to use the PHP language. This is important for how the server is parsed. The second line of code is the simple HTML DOM that contains our auxiliary library. Finally Phpinfo () prints out the current configuration information for PHP to see if the environment is working properly. Be aware that all PHP statements are to be terminated; One of the most common mistakes for beginners is to forget to write a semicolon and output a blank page.

Next, we'll replace the phpinfo with our crawler code. Our task is to get the listings from a website that is being heralded by a TV show. The code is as follows:

<?    include_once(‘simple_html_dom.php‘);    $domain = "http://epg.tvsou.com";    $target_url = "http://epg.tvsou.com/programys/TV_1/Channel_1/W4.htm";    $html = new simple_html_dom();    $html->load_file($target_url);    // 查找channel    $channels = array();    $channels[‘CCTV-1‘] = $target_url;    foreach($html->find(‘div[class=listmenu2] a‘) as $post)    {        $channels[$post->innertext] = $domain.$post->href;    }?>

First, we use $target_url to indicate the crawled address by instantiating the Simple_html_dom object to Load_file load the specified address content. Finally, call find to extract the desired content. The syntax for find is very similar to CSS selectors. The above code gets the channel connection of the program, because the channel link is the link under class=listmenu2 Div, so we put it in the array and pass the linked text (that is, the channel name as the index)

By getting the channel link, we can get further access to the channel's listings, and I'm sure you've been able to put the code behind it. enjoy!

"Go" using PHP to create a basic crawler program