Brother even blockchain training go language crawler writing

Brother even education suggested that users in the consideration of training cycle to effectively combine their current knowledge of the blockchain how much, the purpose of training is a simple cognitive upgrade or take this high-paying employment and so on. Brother even go full stack and the Blockchain training course set to 5.5 months total 22 weeks of learning time, to explain, to help small white users to the transformation of blockchain engineers.

Curriculum System design Framework includes the basic language of the blockchain go language, blockchain backend technology system, blockchain public chain, blockchain distributed application development and other content, as well as to the final interview guidance and project combat. The course was carefully built by Tsinghua University's Microsoft Google Teacher team, which lasted half a year and developed together.

The last time I wrote a reptile in Scala. Recently, at the time of the trip, learning the go language, I used go to transplant that Scala-written crawler, the code is as follows:

Package Main

Import (

"FMT"

"Io/ioutil"

"Net/http"

"RegExp"

)

VAR (

Ptnindexitem = RegExp. Mustcompile (' (. +) ')

Ptncontentrough = RegExp. Mustcompile (' (? s). *

(.*).*`)

Ptnbrtag = RegExp. Mustcompile ('
`)

Ptnhtmltag = RegExp. Mustcompile (' (? s) ')

Ptnspace = RegExp. Mustcompile (' (^\s+) | () `)

)

Func Get (URL string) (content string, StatusCode int) {

RESP, err1: = http. Get (URL)

If err1! = Nil {

StatusCode =-100

Return

}

Defer resp. Body.close ()

Data, ERR2: = Ioutil. ReadAll (resp. Body)

If err2! = Nil {

StatusCode =-200

Return

}

StatusCode = resp. StatusCode

Content = string (data)

Return

}

Type Indexitem struct {

URL string

Title string

}

Func findindex (Content String) (index []indexitem, err Error) {

Matches: = Ptnindexitem.findallstringsubmatch (content, 10000)

index = make ([]indexitem, Len (matches))

For I, Item: = Range Matches {

Index[i] = Indexitem

}

Return

}

Func readcontent (URL string) (content string) {

Raw, StatusCode: = Get (URL)

If StatusCode! = 200 {

Fmt. Print ("Fail to get the raw data from", URL, "\ n")

Return

}

Match: = Ptncontentrough.findstringsubmatch (Raw)

If match! = nil {

Content = Match[1]

} else {

Return

}

Content = ptnbrtag.replaceallstring (content, "\ r \ n")

Content = ptnhtmltag.replaceallstring (content, "")

Content = ptnspace.replaceallstring (content, "")

Return

}

Func Main () {

Fmt. Println (' Get index ... ')

S, StatusCode: = Get

If StatusCode! = 200 {

Return

}

Index, _: = FindIndex (s)

Fmt. Println (' Get contents and write to File ... ')

For _, Item: = Range Index {

Fmt. Printf ("Get content%s from%s and write to file.\n", Item.title, Item.url)

FileName: = Fmt. Sprintf ("%s.txt", Item.title)

Content: = ReadContent (Item.url)

Ioutil. WriteFile (FileName, []byte (content), 0644)

Fmt. Printf ("Finish Writing to%s.\n", FileName)

}

}

There is a certain increase in the number of lines of code than in the Scala version, mainly for the following reasons:

1 Golang attention to code writing specifications, or code format, many places are more fixed, or even more troublesome. For example, even if the execution of the statement is true after a sentence, according to the code specification, but also to write three lines with curly braces, and in Scala and many other languages, one line;

The method provided by the 2 Golang strings package and RegExp package is not particularly useful, especially when compared to Scala, it is more comfortable to use Scala's regular and string processing.

3 The Scala version of the crawler uses the Scala standard library of practical classes and methods, although they are not grammatical composition, but used to feel like a syntactic sugar, there are many methods and functional programming, Golang functional programming has not been carefully studied.

Of course, Golang version of the crawler has an advantage, that is, the speed of the compilation is fast, the execution speed in the present writing does not show the advantages of Golang features Goroutine is not used here, the code will continue to improve in the future.

high-energy early warning, brother even education blockchain live course August continues to hit the hot!

The original price of 1188 Yuan of 12 block chain advanced courses, now only need 1 Yuan!

Also can receive free "Go Language foundation actual combat project development" and "Go language Advanced Combat Project development" textbook Two!!

Limited time limit!! First come first served!!

Http://www.ydma.cn/open/course/24