Java Web Crawler

Java Web Crawler implementation

I remember that when I was just looking for a job, a classmate next door once made a web crawler during the interview. At that time, my admiration was like a flood of rivers and rivers. Later, when searching for images, a large number of test images were required. As a result, the idea of crawling book cover images from Amazon also learned some previous experiences from the Internet, A simple but usable crawler system is implemented.

AD:

I remember that when I was just looking for a job, a classmate next door once made a web crawler during the interview. At that time, my admiration was like a flood of rivers and rivers. Later, when searching for images, a large number of test images were required. As a result, the idea of crawling book cover images from Amazon also learned some previous experiences from the Internet, A simple but usable crawler system is implemented.

Web Crawler is a program for automatically extracting Web pages. It Downloads Web pages from the World Wide Web for search engines and is an important component of search engines. Its basic architecture is shown in:

A traditional crawler obtains the URLs on an initial webpage from the URLs of one or more initial webpages, and continuously extracts new URLs from the current webpage and puts them in the queue, until the system is stopped. For vertical search, focusing on crawlers is more suitable for crawlers that specifically crawl webpages with specific themes.

The core code of this crawler is as follows:

Java code

 
 

  
  
Public void crawl () throws throwable {
  
  
While (continuecrawling ()){
  
  
Crawlerurl url = getnexturl (); // gets the next URL in the queue to be crawled.
  
  
If (URL! = NULL ){
  
  
Printcrawlinfo ();
  
  
String content = getcontent (URL); // obtain the URL text
  
  

  
  
// Focus crawlers only crawl webpages related to the subject content. Here, regular matching is used for simple processing.
  
  
If (iscontentrelevant (content, this. regex1_archpattern )){
  
  
Savecontent (URL, content); // Save the webpage to your local device
  
  

  
  
// Obtain the link in the webpage content and put it in the queue to be crawled
  
  
Collection urlstrings = extracturls (content, URL );
  
  
Addurlstourlqueue (URL, urlstrings );
  
  
} Else {
  
  
System. Out. println (URL + "is not relevant ignoring ...");
  
  
}
  
  

  
  
// Delay prevention from being blocked by the other party
  
  
Thread. Sleep (this. delaybetweenurls );
  
  
}
  
  
}
  
  
Closeoutputstream ();
  
  
}
 
 

The entire function consists of several core methods, including getnexturl, getcontent, iscontentrelevant, extracturls, and addurlstourlqueue. First look at getnexturl:

Java code

 
 

  
  
Private crawler URL getnexturl () throws throwable {
  
  
Crawlerurl nexturl = NULL;
  
  
While (nexturl = NULL )&&(! Urlqueue. isempty ())){
  
  
Crawlerurl = This. urlqueue. Remove ();
  
  

  
  
// Dowehavepermissiontovisit: whether or not you have the permission to access the URL. Friendly crawlers will crawl the URL according to the rules configured in "robot.txt" provided by the website.
  
  
// Isurlalreadyvisited: whether the URL has been accessed. Large search engines usually use bloomfilter to sort the weights. hashmap is used here.
  
  
// Isdepthacceptable: whether to reach the specified depth limit. Crawlers generally take the breadth-first approach. Some websites will build crawler traps (some invalid links are automatically generated to bring crawlers into an endless loop) and use deep restrictions to avoid them
  
  
If (dowehavepermissiontovisit (crawler URL)
  
  
&&(! Isurlalreadyvisited (crawler URL ))
  
  
& Isdepthacceptable (crawler URL )){
  
  
Nexturl = crawler URL;
  
  
// System. Out. println ("next URL to be visited is" + nexturl );
  
  
}
  
  
}
  
  
Return nexturl;
  
  
}
 
 

For more details about robot.txt, refer to the following article:

Http://www.bloghuman.com/post/67/

Getcontent internally uses Apache's httpclient 4.1 to obtain webpage content. The specific code is as follows:

Java code

 
 

  
  
Private string getcontent (crawler URL) throws throwable {
  
  
// The call of httpclient4.1 is different from the previous method.
  
  
Httpclient client = new defaulthttpclient ();
  
  
Httpget = new httpget (URL. geturlstring ());
  
  
Stringbuffer strbuf = new stringbuffer ();
  
  
Httpresponse response = client.exe cute (httpget );
  
  
If (httpstatus. SC _ OK = response. getstatusline (). getstatuscode ()){
  
  
Httpentity entity = response. getentity ();
  
  
If (entity! = NULL ){
  
  
Bufferedreader reader = new bufferedreader (
  
  
New inputstreamreader (entity. getcontent (), "UTF-8 "));
  
  
String line = NULL;
  
  
If (entity. getcontentlength ()> 0 ){
  
  
Strbuf = new stringbuffer (INT) entity. getcontentlength ());
  
  
While (line = reader. Readline ())! = NULL ){
  
  
Strbuf. append (line );
  
  
}
  
  
}
  
  
}
  
  
If (entity! = NULL ){
  
  
Entity. consumecontent ();
  
  
}
  
  
}
  
  
// Mark the URL as accessed
  
  
Markurlasvisited (URL );
  
  
Return strbuf. tostring ();
  
  
}
 
 

For vertical applications, data accuracy is often more important. The main feature of a focused crawler is to collect only theme-related data, which is the role of the iscontentrelevant method. The classification prediction technology may be used here, instead of regular matching for simplicity. The main code is as follows:

Java code

 
 

  
  
Public static Boolean iscontentrelevant (string content,
  
  
Pattern regexppattern ){
  
  
Boolean retvalue = false;
  
  
If (content! = NULL ){
  
  
// Whether the regular expression conditions are met
  
  
Matcher M = regexppattern. matcher (content. tolowercase ());
  
  
Retvalue = M. Find ();
  
  
}
  
  
Return retvalue;
  
  
}
 
 

Extracturls is mainly used to obtain more URLs from webpages, including internal links and external links. The Code is as follows:

Java code

 
 

  
  
Public list extracturls (string text, crawler URL ){
  
  
Map urlmap = new hashmap ();
  
  
Extracthttpurls (urlmap, text );
  
  
Extractrelativeurls (urlmap, text, crawler URL );
  
  
Return new arraylist (urlmap. keyset ());
  
  
}
  
  

  
  
// Process external links
  
  
Private void extracthttpurls (MAP urlmap, string text ){
  
  
Matcher M = httpregexp. matcher (text );
  
  
While (M. Find ()){
  
  
String url = M. Group ();
  
  
String [] terms = URL. Split ("a href = \"");
  
  
For (string term: Terms ){
  
  
// System. Out. println ("term =" + term );
  
  
If (term. startswith ("HTTP ")){
  
  
Int Index = term. indexof ("\"");
  
  
If (index> 0 ){
  
  
Term = term. substring (0, index );
  
  
}
  
  
Urlmap. Put (term, term );
  
  
System. Out. println ("hyperlink:" + term );
  
  
}
  
  
}
  
  
}
  
  
}
  
  

  
  
// Process internal links
  
  
Private void extractrelativeurls (MAP urlmap, string text,
  
  
Crawlerurl ){
  
  
Matcher M = relativeregexp. matcher (text );
  
  
URL texturl = crawler URL. geturl ();
  
  
String host = texturl. gethost ();
  
  
While (M. Find ()){
  
  
String url = M. Group ();
  
  
String [] terms = URL. Split ("a href = \"");
  
  
For (string term: Terms ){
  
  
If (term. startswith ("/")){
  
  
Int Index = term. indexof ("\"");
  
  
If (index> 0 ){
  
  
Term = term. substring (0, index );
  
  
}
  
  
String S = "http: //" + host + term;
  
  
Urlmap. Put (S, S );
  
  
System. Out. println ("relative URL:" + S );
  
  
}
  
  
}
  
  
}
  
  

  
  
}
 
 

In this way, a simple web crawler program is built and can be tested using the following program:

Java code

 
 

  
  
public static void main(String[] args) {     
  
  
    try {     
  
  
        String url = "http://www.amazon.com";     
  
  
        Queue urlQueue = new LinkedList();     
  
  
        String regexp = "java";     
  
  
        urlQueue.add(new CrawlerUrl(url, 0));     
  
  
        NaiveCrawler crawler = new NaiveCrawler(urlQueue, 100, 5, 1000L,     
  
  
                regexp);     
  
  
        // boolean allowCrawl = crawler.areWeAllowedToVisit(url);     
  
  
        // System.out.println("Allowed to crawl: " + url + " " +     
  
  
        // allowCrawl);     
  
  
        crawler.crawl();     
  
  
    } catch (Throwable t) {     
  
  
        System.out.println(t.toString());     
  
  
        t.printStackTrace();     
  
  
    }     
  
  
}    
 
 

Of course, you can give it more advanced functions, such as multithreading, more intelligent focus, and indexing with Lucene. In more complex cases, you can consider using some open-source spider programs, such as nutch or heritrix, which will not be discussed in this article.

Reprinted: http://developer.51cto.com/art/201103/248141.htm