PHP implementation of simple crawler development, PHP implementation Crawler _php Tutorial

PHP implementation of simple crawler development, PHP implementation crawler

Sometimes because of work, their own needs, we will go to browse different sites to get the data we need, so the crawler came into being, below is my development of a simple crawler after and encountered problems.

To develop a crawler, first you need to know what your reptile is for. I'm going to use a different website to find a specific keyword article and get a link to it so I can read it quickly.

According to personal habits, I first have to write an interface, to clarify the idea.

1, go to different websites. Then we need a URL input box.
2, search for specific keywords of the article. Then we need an article title entry box.
3, get the article link. Then we need a display container for search results.

    Article URL crawl          article title              Site URL           Crawl       Article URL      
   

Directly on the code, and then add some of your own style adjustments, the interface is complete:

Then next is the implementation of the function, I use PHP to write, the first step is to get the HTML code of the website, the way to get the HTML code is also a lot, I do not introduce, here with curl to get, the site URL can be obtained HTML code:

Private Function get_html ($url) {  $ch = Curl_init ();  $timeout = ten;  curl_setopt ($ch, Curlopt_url, $url);  curl_setopt ($ch, Curlopt_returntransfer, 1);  curl_setopt ($ch, curlopt_encoding, ' gzip ');  curl_setopt ($ch, Curlopt_useragent, ' mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/34.0.1847.131 safari/537.36 ');  curl_setopt ($ch, Curlopt_connecttimeout, $timeout);  $html = curl_exec ($ch);  return $html; }

Although get the HTML code, but soon you will encounter a problem, that is the coding problem, which may let your next match reactive, we here unify the resulting HTML content into UTF8 encoding:

$coding = mb_detect_encoding ($html), if ($coding! = "UTF-8" | |!mb_check_encoding ($html, "UTF-8")) $html = Mb_convert_ Encoding ($html, ' utf-8 ', ' gbk,utf-8,ascii ');

Get the Web site's HTML, to get the URL of the article, then the next step is to match all the a tag under the page, need to use regular expression, after many tests, and finally get a more reliable regular expression, regardless of a tag structure is more complex, as long as the a tag is not spared: (the most important step)

$pattern = ' |] *> (. *) |isu ';p reg_match_all ($pattern, $html, $matches);

The result of the match is in $matches, which is probably one of these multidimensional groups:

Array (2) {  [0]=>  Array (*) {   [0]=>  string (*) "full A-label"  ...  } [1]=> Array (*) {  [0]=>  string (*) "The contents of the a tag corresponding to the subscript above"}}

As long as you can get this data, the other can be fully operational, you can traverse the group, find you want a tag, and then get the corresponding properties of a tag, how to operate it, the following recommended a class, let you more convenient operation a tag:

$dom = new DOMDocument (); @ $dom->loadhtml ($a);//$a is some of the above obtained a tag $url = new Domxpath ($dom); $hrefs = $url->evaluate ('//a '); for ($i = 0; $i < $hrefs->length; $i + +) {  $href = $hrefs->item ($i);  $url = $href->getattribute (' href '); This gets the href attribute of the A tag}

Of course, this is just one way, you can also use regular expressions to match the information you want, to play the data new tricks.

Get and match the results you want, the next step of course is to return to the front end to show them, the interface is written, and then the front end with JS to get the data, with jquery dynamic add content display:

var website_url = ' Your interface address '; $.getjson (website_url,function (data) {if (data) {  if (Data.text = = ') {   $ (' #article_ URL '). html ('
No link to this article
');   return;  }  var string = ';  var list = Data.text;  for (var j in list) {    var content = list[j].url_content;    for (var i-in content) {     if (content[i].title! = ") {      string + = ' + ' +       '[' + List[j].website.web_name + ' + ' + ' + content[i].title + ' + ';}}   }  $ (' #article_url '). HTML (string);});

On the final:

Articles you may be interested in:

• PHP IIS Log analysis search engine crawler log Program
• PHP displays different content to visitors and crawlers
• A lightweight and simple crawler implemented by PHP
• PHP implementation of simple crawler methods
• PHP code to implement crawler records--super-works
• PHP Crawler's million-level knowledge of user data crawling and analysis
• Php+html+javascript+css for simple crawler development

http://www.bkjia.com/PHPjc/1117098.html www.bkjia.com true http://www.bkjia.com/PHPjc/1117098.html techarticle PHP Implementation of simple crawler development, PHP implementation of the crawler sometimes because of work, their own needs, we will go to browse different sites to obtain the data we need, so the crawler came into being, ...

