PHP uses the Simple_html_dom class to get page content and act as a crawler

PHP script to play the role of reptiles, may be the first time you think it might be a regular, personal rules of the regular can not remember, it is difficult to start, today there is a need to crawl a site some store information

Inadvertently on the Internet to see a better class library called: Simple_html_dom

Github:https://github.com/samacs/simple_html_dom

The most important step: you need to understand the structure of other people's website, and know which tab to start is the data you want

Below is a demonstration of the process.

The implementation process I took three steps.

1, the store information latitude and longitude, name and other important information first inserted into the local surface

[PHP]View PlainCopy

2. Set_time_limit (0);
4. $host = ' 127.0.0.1 ';
6. $user = ' root ';
8. $user _pwd = ";
10. $database = ' Dataname ';
12. $conn = mysql_connect ($host,$user,$user _pwd) or Die (' SSS ');
14. mysql_select_db ($database,$conn) or Die (' dddd ');
16. mysql_query (' Set names UTF8 ');
22. Include ('./simple_html_dom-master/simple_html_dom.php ');
26. $url = ' site URL to crawl ';
28. $html = file_get_html ($url);
30. $n = 1;
32. foreach ($html->find (' li[data-counts=0] ') as $e) {
34. $storeid = $e->storeid;
36. $star = $e->level.  0 ';
38. $work _time = $e->time;
40. $mapx = $e->mapx;
42. $mapy = $e->mapy;
44. $nickname = $e->mapname;
46. $mapadd = $e->mapadd;
48. $maptel = $e->maptel;
50. $time = date (' y-m-d h:i:s ');
52. $query = "INSERT into ' store ' (' StoreID ', ' star ', ' Work_time ', ' longitude ', ' latitude ', ' create_time ', ' nickname ', ' Address ', ' tel ')
54. VALUES ($storeid,' ". $star." ',' ". $work _time." ', ' ".$mapx." ', ' ". $mapy." ', ' ". $time." ', ' " ". $nickname." ',' ". $mapadd." ',' ". $maptel." ') ";
56. $res = mysql_query ($query);
58. //echo $query; exit ();
62. if ($res) {
64. echo ' successfully imported into the section '.   $n.' stores <br> ';
66. $n + +;
68. }else{
70. Die (' failure <br> ');
72. }
74. }

2, jump into the site of another page to get store logo image

[PHP]View PlainCopy

2. $query = "Select StoreID from store ORDER by id DESC";
6. $row = mysql_query ($query);
10. while ($rows = mysql_fetch_array ($row)) {
12. $url = ' http://Others ' site domain name/'. $rows [' StoreID '].  Jhtml ';
14. $html = file_get_html ($url);
16. foreach ($html->find (' div.onlyonepic ') as $e) {
18. //Get the SRC attribute of img
20. $img = $e->firstchild ()->src;
22. //Save remote picture to local
24. $content = file_get_contents ($img);
26. file_put_contents ('./store/'). $rows [' StoreID '].  JPEG ', $content);
28. }
30. }

3, update the table of the corresponding Store logo field

[PHP]View PlainCopy

2. $query = "Select StoreID from store ORDER by id DESC";
4. $row = mysql_query ($query);
6. $n = 1;
8. while ($rows = mysql_fetch_array ($row)) {
10. $img = "https://my own site domain name/". $rows [' StoreID '].".  JPEG ";
12. $sql = "UPDATE store set Img_url= '". $img. "' Where storeid=."  $rows [' StoreID '];
14. $res = mysql_query ($sql);
16. if ($res) {
18. echo ' successfully updated section '.   $n.' stores <br> ';
20. $n + +;
22. }else{
24. echo ' failure ';
26. }
28. }

OK, the function is realized, but there is no more in-depth understanding of the other functions of this class library, here is just a record, convenient for later when needed to use

PHP uses the Simple_html_dom class to get page content and act as a crawler