Learning notes for ultra-small open-source crawler Crawlers

Document directory

• 1. url splicing (urlutils. Java)
• 2. encoding of the webpage source code
• 3. Miscellaneous

Recently, I want to write a small crawler framework. Unfortunately, zero has no experience in writing a framework. Therefore, it is necessary to find an existing framework for reference. Google found that the crawler is the best reference for the framework to be written. The crawler is a simple crawler framework that implements the common part of the crawler, such as URL splicing and webpage encoding, so that users can focus on extracting webpage content (original article: crawler is a simple Java Web Crawler/spider/Joe or any other name you want to call it. the main goal is to abstract that boring and error-prone code from your codebase and let you focus on crawling the site .). When I read the source code of crawler, I found that there are many areas worth learning from, so I recorded them one by one.

1. url splicing (urlutils. Java)

The Uniform Resource Locator extracted from a webpage may be complete or relative. However, when requesting a webpage, you can only use the complete URL. Therefore, you must change the relative URL to a complete URL. Before conversion, you should first understand the URL Syntax:
Scheme: // domain: Port/path? QUERY_STRING # fragment_id
Scheme indicates various protocols, such as HTTP, https, and FTP. Domain indicates the domain name or IP address of the server. Port indicates the port number. Path indicates the directory where the file is located; QUERY_STRING is the input data of a program on the server. fragment_id refers to a certain position on the page. It allows users to automatically locate a specified position when opening a webpage.

According to this syntax, the splicing steps are as follows:
A. Split the URL into five parts, scheme, domain, path, QUERY_STRING, and fragment_id based on the features of each part (for example, scheme appears at the beginning ).
B. Check the URL in the order of scheme, domain, path, QUERY_STRING, and fragment_id. If a part is missing, add it.
C. If "& amp;" appears in the URL, replace it "&"
D. encode the URL as a UTF-8 to prevent the extracted URL from containing Chinese characters.

2. encoding of the webpage source code

There are two methods to detect the encoding of the webpage source code. One is to analyze the meta information in the webpage source code, such as contenttype, to obtain the encoding. The other is to use statistical and heuristic methods to detect web page source code. Because the contenttype of some web pages does not contain encoding information, the first method may not work. In this case, the second method is required. The icu4j library of IBM implements the second method. The usage is as follows:

/*** Get the webpage Code * @ Param pagecontent the webpage content * @ return the webpage encoding method */public static string getpagecharset (byte [] pagecontent) {charsetdetector detector = new charsetdetector (); detector. settext (pagecontent); charsetmatch match = detector. detect (); Return match. getname ();}

3. Miscellaneous

In addition to the above two main aspects, there are also some minor aspects, including:

A. Set all cookies before using httpclient.

B. Check whether the MIME type is desired before processing the webpage.

C. Use org. Apache. log4j. logger to record various running information for emergency purposes.

D. Create an Enum class to summarize various HTTP Code states, such as the code in the OK status from 200 to 299, And the redirection status from 300 to 399.

E. Use the final keyword for the variable when it is determined that it will not change the value of the variable.