This example describes the web crawler approach to the go implementation. Share to everyone for your reference. The specific analysis is as follows:
This uses the Go Concurrency feature to execute the web crawler in parallel.
Modify the Crawl function to crawl URLs in parallel and ensure that they are not duplicated.
Copy Code code as follows:
Package Main
Import (
"FMT"
)
Type Fetcher Interface {
Fetch returns the body content of the URL and puts the URL found on the page in a slice.
Fetch (URL string) (body string, URLs []string, err Error)
}
Crawl uses Fetcher to crawl the page recursively from a URL until the maximum depth is reached.
Func Crawl (URL string, depth int, fetcher fetcher) {
TODO: Crawl URLs in parallel.
TODO: Do not repeat crawl pages.
The following two scenarios are not implemented:
If depth <= 0 {
Return
}
Body, URLs, err: = Fetcher. Fetch (URL)
If Err!= nil {
Fmt. PRINTLN (ERR)
Return
}
Fmt. Printf ("Found:%s%q\n", URL, body)
For _, U: = Range URL {
Crawl (U, depth-1, fetcher)
}
Return
}
Func Main () {
Crawl ("http://golang.org/", 4, Fetcher)
}
Fakefetcher is a fetcher that returns a number of results.
Type Fakefetcher Map[string]*fakeresult
Type Fakeresult struct {
Body string
URLs []string
}
Func (f *fakefetcher) Fetch (URL string) (string, []string, error) {
If res, OK: = (*f) [url]; OK {
Return res.body, Res.urls, nil
}
Return "", Nil, FMT. Errorf ("Not Found:%s", URL)
}
Fetcher is the fakefetcher after filling.
var fetcher = &fakefetcher{
"http://golang.org/": &fakeresult{
"The Go Programming Language",
[]string{
"http://golang.org/pkg/",
"Http://golang.org/cmd/",
},
},
"http://golang.org/pkg/": &fakeresult{
"Packages",
[]string{
"http://golang.org/",
"Http://golang.org/cmd/",
"Http://golang.org/pkg/fmt/",
"Http://golang.org/pkg/os/",
},
},
"http://golang.org/pkg/fmt/": &fakeresult{
"Package FMT",
[]string{
"http://golang.org/",
"http://golang.org/pkg/",
},
},
"http://golang.org/pkg/os/": &fakeresult{
"Package OS",
[]string{
"http://golang.org/",
"http://golang.org/pkg/",
},
},
}
I hope this article will help you with your go language program.
