Golang Native Crawler Simple crawler implement non-dependent third party Package library easy to understand technology principles (II.)

This is a creation in Article, where the information may have evolved or changed.

Previous review: Golang native crawler simple crawler implementation of non-reliance on third-party package library easy to understand technology principles (I.)

This article starts: Golang Native crawler simple crawler implementation of non-reliance on third-party package library easy to understand the principle of technology (II)

After the first successful execution of the program, we have successfully obtained the link address provided by the source page. All we have to do now is to crawl through the list of links with a timer.

The next step is to crawl through addresses in the address list, remove the practice levels that have been crawled, and record the new valid links to the address lists.

Take a look at our main function:

func main() {      if checkFile("./data/", "url.txt").Size() == 0 {          fistStart()          main()      } else {          Timer()      }  }  

The Firststart function above (the first crawl) has been executed, then the main function is called again, that is, the execution of a judgment, but because our url.txt already has 12 URL address, so this time will execute the timer function.

Timer function We wrote a timer to prevent the program crashes or network crashes, so I set up here 1 seconds to execute once, in fact, there is no need to do this, 3-8 times a second is no big problem (local), if placed on the server, Then you have to look at your own server configuration and bandwidth configuration consider, as appropriate,

Take a look at the timer function:

Func timer ()  {      t := time. Newtimer (time. second * 1)       <-t.c      fmt. Print ("\n\n\n performs crawl-catch \ n")       f, _ := os. OpenFile ("./data/url.txt",  os. O_create|os. O_append|os. o_rdwr, 0666)       file, _ := ioutil. ReadAll (f)       pagecont, _ := pagevisit (strings. Split (String (file),  "\ n") [0])       if checkregexp (Checkregexp (Pagecont ,  regtitle, 0). (string),  regchecktitle, 0). (string)  !=  ""  {          fmt. Print (Checkregexp (Checkregexp (pagecont, regtitle, 0). string),  regchecktitle, 0). (string))           fmt. Print ("\ n valid content  => " &NBSp;+ checkregexp (pagecont, regtitle, 0). (string))       }      fmt. Print ("\ n" to crawl URL total " + strconv. Itoa (Len (strings). Split (String (file),  "\ n"))-1)  +  " => "  + strings. Split (String (file),  "\ n") [0] +  "\ n")       delfirsttext ("./data/ Url.txt ")       timer ()   }  

Emm ... It's no exaggeration to say that I look at my code with a little difficulty.

The above code creates a timer with a time of one second. At first I must open the Url.txt file, because it is to do the deletion and add the operation, so the open mode is read-write append.

pageCont, _ := pageVisit(strings.Split(string(file), "\n")[0])  

This sentence is to get the first link address in url.txt, we have to determine the link content is not what we want, so I used a previous package of a regular check function.

Checkregexp function:

func checkRegexp(cont string, reg string, style int) (result interface{}) {      check := regexp.MustCompile(reg)      switch style {      case 0:          result = check.FindString(cont)      case 1:          result = check.FindAllString(cont, -1)      default:          result = check.FindAll([]byte(cont), -1)      }      return  }

The use of regular and Regtitle, Regchecktitle

var (      // regHref       = `((ht|f)tps?)://[w]{0,3}.baidu.com/link\?[a-zA-z=0-9-\s]*`      regTitle      = `<title[\sa-zA-z="-]*>([^x00-xff]|[\sa-zA-Z=-：|，？"])*</title>`      regCheckTitle = `(为什么|怎么)*.*([G|g][O|o][L|l][A|a][N|n][G|g]).*(怎么|实现|如何|为什么).*`  )

Regtitle is to match the real title in the code, because some sites to prevent reptiles, do some false headlines to confuse, but these small tricks are easy to solve, this regtitle enough to block out the 70% false title.

Anyway crawler is to and each big website wits to fight/manual funny

Regchecktitle is to filter out this URL is not what I want, so I simply wrote a string of regular. This string of regular meaning is mainly the title with why, how to wait for keywords, and then the title must have Golang or go existence, such content is basically what I want.

To determine the code snippet:

if checkRegexp(checkRegexp(pageCont, regTitle, 0).(string), regCheckTitle, 0).(string) != "" {      fmt.Print(checkRegexp(checkRegexp(pageCont, regTitle, 0).(string), regCheckTitle, 0).(string))      fmt.Print("\n有效内容 => " + checkRegexp(pageCont, regTitle, 0).(string))  } 

If the match is not, then skip this link, anyway, it's not what I want ~

DelFirstText("./data/url.txt")

func delfirsttext (file string)  {       var text =  "" &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;F,&NBSP;_&NBSP;: = os. OpenFile (File, os. O_rdwr|os. o_create, 0666)       files, _ := ioutil. ReadAll (f)       var ss = strings. Split (files),  "\ n")       for i := 1; i <  len (ss) -1; i++ {          text +=  ss[i] +  "\ n"       }      defer f. Close ()       ioutil. WriteFile (File, []byte (text),  0666)       fmt. Print ("\ n \ nyou Delete the address  => "  + ss[0])   }   

Then this paragraph means to delete this link address, if there is not a paragraph, your crawler will take the trouble to crawl the first link address, can continue to crawl your IP is handled by the server security program

Someone should find out, and then what? How not to put things into storage, how not to crawl new links.

Emm ... Bo Master recently a little busy, this paragraph has not written, but these things have been the basic principles of the crawler to tell off, in fact, is very simple right, is to initiate an HTTP request, and then through the regular match to the content they want, then do the subsequent storage or inject a fresh link address, so that the program has been running on good.

Run it:

Well, in that case, the effective content is filtered out.

Then if you want to extract the contents of the article, only need a very simple regular, the processing steps here can actually write a separate function to invoke. But the blogger has not written yet, and may be in the third article, if the volume of access broken 2k

In order to prevent some of the details of the code is not affixed, I put main.go code, you can also go to the bottom of the article link to download the code example of the whole program, if there is a problem can call me, QQ in the last article, will not be repeated affixed.

Main.go:

package main    import  (       "FMT"         "io"        "Io/ioutil"         "Net/http"        "OS"        "RegExp"         "StrConv"        "strings"         "Time"   )     var  (      //  reghref       =  ' ((ht|f) TPS?):/ /[w]{0,3}.baidu.com/link\? [a-za-z=0-9-\s]* '       regTitle      =  ' < Title[\sa-za-z= "-]*> ([^x00-xff]|[ \sa-za-z=-:|, huh? "]) *</title> '       regCheckTitle =  ' (why | how) *. * ([g|g][o|o][l|l][ A|A][N|N][G|G]). * (How to | implement | how | why). *   )     func main ()  {       if checkfile ("./data/",  "Url.txt"). Size ()  == 0 {          fiststart ()            main ()       } else  {          timer ()        }  }    func timer ()  {      t :=  time. Newtimer (time. second * 1)       <-t.c      fmt. Print ("\n\n\n performs crawl-catch \ n")       f, _ := os. OpenFile ("./data/url.txt",  os. O_create|os. O_append|os. o_rdwr, 0666)       file, _ := ioutil. ReadAll (f)       pagecont, _ := pagevisit (strings. Split (String (file),  "\ n") [0])       if checkregexp (Checkregexp (pagecont, regtitle, 0). ( string),  regchecktitle, 0). (string)  !=  ""  {          fmt. Print (Checkregexp (Checkregexp (pagecont, regtitle, 0). string),  regchecktitle, 0). (string))           fmt. Print ("\ n valid Content  => "  + checkregexp (pagecont, regtitle, 0). String))       }      fmt. Print ("\ n" to crawl URL total " + strconv. Itoa (Len (strings). Split (String (file),  "\ n"))-1)  +  " => "  + strings. Split (String (file),  "\ n") [0] +  "\ n")       delfirsttext ("./data/ Url.txt ")       timer ()   }    func fiststart ()  {      var num int      url :=  "Http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1 &tn=39042058_20_oem_dg&wd=golang%e5%ae%9e%e7%8e%b0&oq=golang%2520%25e5%2588%25a0%25e9%2599%25a4% 25e6%2595%25b0%25e7%25bb%2584&rsv_pq=d9be28ec0002df1b&rsv_t=8017gwpslphdmkilzq1stc04evpuaelep90nim% 2bk5prh5r9o57nhmo8gaxm1ttsoo%2fvtjj%2b98%2fsc&rqlang=cn&rsv_enter=1&inputt=3474&rsv_sug3=16 &rsv_sug1=11&rsv_sug7=100&rsv_sug2=0&rsv_sug4=4230 "      resp,  _ := http. Get (URL)       defer resp. Body.close ()       body, _ := ioutil. ReadAll (resp. Body)       reg := regexp. Mustcompile (' (ht|f) TPS?):/ /[w]{0,3}.baidu.com/link\? [a-za-z=0-9-\s]* ')       f, _ := os. OpenFile ("./data/url.txt",  os. O_create|os. O_append|os. o_rdwr, 0666)       defer f.close ()       for _, d  : = range reg. Findallstring (String (body),  -1)  {          ff,  _ := os. OpenFile ("./data/url.txt",  os. o_rdwr, 0666)           file, _ := ioutil . ReadAll (FF)           dd := strings. Split (d,  "")           dddd :=  ""            for _, ddd := range dd {               if ddd ==  "?"  {                   ddd =  ' \? ' &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&Nbsp;       }               dddd += ddd          }           if checkregexp (String (file), dddd,  0). (string)  ==  ""  {               io. WriteString (f, d+ "\ n")                fmt. Print ("\ n Collect Address:"  + d +  "\ n")                num++          }           // fmt. Print (string (file))           ff. Close ()       }      fmt. Print ("\ nthe first collection of network addresses:"  + strconv. Itoa (Len (Reg. Findallstring (String (body),  -1)))  +  "\ n")       fmt. Print ("\ nthe network address number:"  + strconv. Itoa (num))       fmt. Print ("\ n \ nthe First Storage success!) \ n ")   }    func pagevisit (url string)   (page string,  Body []byte)  {      resp, _ := http. Get (URL)       defer resp. Body.close ()       body, _ = ioutil. ReadAll (resp. Body)       page = string (body)        Return  }    func checkfile (dir string, file string)  os . Fileinfo {      list, _ := ioutil. ReadDir (dir)       for _, info := range lisT {          if info. Name ()  == file {               return info          }       }      return list[0]  }    func  savefile (file string, cont string)  {      f, _  := os. OpenFile (File, os. O_rdwr|os. O_append|os. o_create, 0666)       defer f.close ()        io. WriteString (F, cont)   }    func checkregexp (cont string, reg  string, style int)   (result interface{})  {       Check := regexp. Mustcompile (REG)       switch style {      case 0:           result = check. FindString (cont)       case 1:           result = check. Findallstring (cont, -1)       default:           result = check. FindAll ([]byte (cont),  -1)       }      return   }    func delfirsttext (file string)  {       var text =  ""       f, _ := os. OpenFile (File, os. O_rdwr|os. o_create, 0666)       files, _ := ioutil. ReadAll (f)       var ss = strings. Split (string (files),  "\ n")      for i := 1; i < len (ss) -1; i++  {          text += ss[i] +  "\ n"        }      defer f.close ()        ioutil. WriteFile (File, []byte (text),  0666)       fmt. Print ("\ n \ nyou Delete the address  => "  + ss[0])   }

Attached code example: https://download.csdn.net/download/superwebmaster/10415730