A Tour of Go-exercise:web Crawler

This is a creation in Article, where the information may have evolved or changed.

A Tour of Go

Exercise:web Crawler

In this exercise you'll use the Go ' s concurrency features to parallelize a web crawler.

Modify the Crawl function to fetch URLs in parallel without fetching the same URL twice.

Package Mainimport ("FMT") type Fetcher interface {//Fetch returns the body of URL and//a slice of URLs found on this pag E.fetch (URL string) (body string, URLs []string, err Error)}//Crawl uses fetcher to recursively crawl//pages starting WI Th URL, to a maximum of Depth.func Crawl (URL string, depth int., Fetcher Fetcher, Out Chan string, Mutuex Chan map[string]b OOL, End chan bool) {//Todo:fetch URLs in parallel.//Todo:don ' t Fetch the same URL twice.//this implementation doesn ' t do either:if depth <= 0 {end <-truereturn}body, URLs, err: = Fetcher. Fetch (URL) if err! = Nil {out <-err. Error () End <-truereturn}fmt. Printf ("Found:%s%q\n", URL, body) visited: = <-mutuexsubend: = Make (chan bool) I: = 0for _, u: = Range URLs {if!visit Ed[u] {Visited[u] = Truei ++go Crawl (U, depth-1, Fetcher, out, Mutuex, subend)}}mutuex <-visitedfor; i = = 0; i--{<-subend}end <-truereturn}func Main () {out: = Do (chan string) Mutuex: = Make (chan map[string]bool) visited: = maKe (map[string]bool) End: = Make (chan bool) visited["http://golang.org/"] = Truego Crawl ("http://golang.org/", 4, Fetcher , out, Mutuex, end) Mutuex <-visitedfor {Select {case t:= <-out:fmt. PRINTLN (t) case <-end:return}}}//Fakefetcher are fetcher that returns canned Results.type fakefetcher Map[string]*fake Resulttype fakeresult struct {body Stringurls []string}func (f *fakefetcher) Fetch (URL string) (string, []string, Erro R) {If res, OK: = (*f) [url]; OK {return res.body, Res.urls, Nil}return "", Nil, FMT. Errorf ("Not Found:%s", url)}//Fetcher is a populated fakefetcher.var Fetcher = &fakefetcher{"http://golang.org/": &A Mp;fakeresult{"The Go programming Language", []string{"http://golang.org/pkg/", "http://golang.org/cmd/",},}, "http ://golang.org/pkg/": &fakeresult{" Packages ", []string{" http://golang.org/"," http://golang.org/cmd/","/HTTP/ golang.org/pkg/fmt/"," http://golang.org/pkg/os/",},}," http://golang.org/pkg/fmt/": &fakeresult{" Package FMT " , []string{] Http://golang.org/"," http://golang.org/pkg/",},}," http://golang.org/pkg/os/": &fakeresult{" Package OS ", []string{]/HTTP/ golang.org/"," http://golang.org/pkg/",},},}