Java Crawler Understanding

1, download (page HTML), request header (such as user-agent) processing, redirection, timeout, agent, retry policy, etc.

1) Using URL implementations in the JDK

HttpURLConnection con = url.openconnection ()

2) using the Apache project HC Implementation http://hc.apache.org/

HttpResponse response = Httpclient.execute (HttpRequest)

2. Parsing (link to crawl + structured data)

1) Regular, pattern class

2) jsoup,http://www.oschina.net/p/jsoup,http://jsoup.org/

3) xpath,http://www.w3school.com.cn/xpath/index.asp

3. Storage (original HTML + structured data)

1) file system for retrieval (LUCENE,HADOOP)

2) database

3) URL score?

4. Dispatch

1) URL priority to crawl

2) go to Heavy

3) concurrency

4) distributed

Java Crawler Understanding