[Wbia 1.1] heritrix crawls webpage Information

[Wbia 1] indicates Part 1 of Web Based Information Architecture Job 1. readers who have searched this log can ignore it directly.

I have a relatively low understanding of heritrix. I hope this article will help youProgramApe is helpful. If you have any errors, please leave a message to correct them. Thank you very much.

Heritrix is an open-source crawler that can freely configure the crawling process. Heritrix can obtain complete, accurate, and in-depth copy of site content. Including obtaining images and other non-text content. Capture and store related content. Do not reject content visitors, and do not modify the content on the page. Re-crawling does not replace the previous one with the same URL. Crawlers can start, monitor, and adjust Web user interfaces to flexibly define URLs to be obtained.

I. About installation and operation (part of content is transferred from http://blessed24.iteye.com/blog/969777)

1. Download: Go to www.sourceforge.netto search for heritrix. then download heritrix-1.14.4-rc1.zip.

2. Installation

(1).decompress heritrix-1.14.4-rc1.zip. Assume that the folder is decompressed to the root directory of the C drive and the decompressed folder is named heritrix.

(2) Enter c: \ heritrix \ conf to copy the jmxremote. Password. template file to c: \ heritrix and rename the file jmxremote. Password,

Then modify the content to the following:

Monitorrole @ 123456789 @-> monitorrole admin (the password is set between @ and @, followed by-> User Role username)

Controlrole @ 123456789 @-> controlrole Shi

Set the file jmxremote. Password to read-only.

(3) Go to c: \ heritrix \ conf, open the file heritrix. properties, and modify the key-value values.

Heritrix. cmdline. Admin = admin: 770629 (User name: password)

Heritrix. cmdline. Port = 8080 (the default port number of the heritrix server is 8080, so you do not need to change the port if it is not used)

(4) Open CMD and switch the directory to C: \ heritrix \ bin.

Run the following command: heritrix -- admin = admin: 123456789.

As heritrixis an open source software, if you need to change it, you can search and download its source code heritrix-1.14.0-rc1-src.zip, and see the http://blessed24.iteye.com/blog/969777 for eclipse configuration. Heritrix has flexible configurations and does not need to modify the source code.

Ii. Pre-capturing Configuration

1. after completing "1", you can use a browser to access the heritrix interactive interface. If it is configured on the local machine, visit the URL http: // 127.0.0.1: 8080. If it is configured on the server, the IP address and port number of the server.

2. Start a simple crawling task (part of the content is transferred from the http://www.iteye.com/topic/84206)

(1) In the upper navigation bar of Wui, select "jobs". The first item displayed is "create new job", and the fourth item is "with defaults ". The first two names and descriptions of the input items are random. seeds is very important: http://www.baidu.com/#the last one.

(2) Select "modules" below to go to the module configuration page (heritrix's extension functions are implemented through the module concept and can implement its own modules to complete the desired functions ). The first item"Select crawl Scope"Use the default" org. archive. crawler. deciderules. decidingscope ". The last three items"Select writers"Delete the default" org. archive. crawler. writer. arcwriterprocessor ", and then add" org. archive. crawler. writer. unzip writerprocessor ", so that the page captured during task execution will be placed in the local directory structure as an image instead of generating the arc archive file.

(3) Select "submodules" on the right of "modules". In the first item"Crawl-order->Scope->Decide-Rules->Rules"Delete" acceptiftranscluded "(Org. archive. crawler. deciderules. transclusiondeciderule. Otherwise, when the HTTP request returns 301 or 302, heritrix crawls webpages in other domains.

(4) Select "Settings" in the second navigation bar of Wui To Go To The Job configuration page. Modify the following items:HTTP-headersUnder User-Agent and from, replace "project_url_here" and "contact_email_address_here" with your own content ("project_url_here" must begin with "http)

(5) Select the rightmost "submit job" in the second navigation bar of Wui"

(6) Select "console" for the first item in the navigation bar of the first line of Wui, and click "start" to capture the task, the length of time is related to the network condition and the depth of the captured website.

3. Personalized Configuration

(1). Domain Name Regular Expression

Our task is to capture the webpage of PKU deep research institute and its affiliated departments, institutes, and centers. There is a problem in capturing with the above method. You cannot capture the systems, institutes, and Centers under a domain name other than pkusz.edu.cn. If you open the domain name restrictions, you will capture the scope of Peking University and Shenzhen Institute. In this case, you need to use a regular expression for filtering. In modules, select org. archive. crawler. deciderules. decidingscope, and then selectOrg. archive. crawler. deciderules. matcheslistregexpdeciderule,Delete all others. Add five regular expressions to Regexp-list in setting ,. * szpku. *. * pnbs. PKU. *. * STL. PKU. *. * pkusz. *. * szcie. PKU. * Note that the wildcard character of the Java regular expression is. * Instead *. In addition, the selection of the module and submodule affects the content in setting. The corresponding module is selected for the corresponding setting.

(2). restrict the types of captured files

Most of the time, we are concerned about the text of web pages, so we do not want to capture a bunch of images, videos, compressed packages or anything, so we need to limit it. In submodule, addOrg. archive. crawler. deciderules. matchesfilepatterndeciderule,For the meaning of each submodule, you can click the question mark next to it. This submodule is used to acceptNoSeveral types of files. In setting, accept all indicates that allNoThe following types of files:

Images:. BMP,. GIF,. JP (e) g,. PNG,. tif (f)

Audio:. mid, MP2,. MP3,. MP4,. wav

Video:. Avi,. mov,. MPEG,. Ram,. rm,. smil,. wmv

Miscellaneous:. Doc,. pdf,. ppt,. SWF

In this way, the captured image will be much smaller.

(3). Modify the encoding type

Garbled characters may appear when capturing Chinese characters. You need to modify the encoding format. Search for encoding in setting and fill in the UTF-8 in the air.

Iii. capture process

The whole crawling process lasted for more than three days, but most of the webpages were captured in the first night. Because crawlers are constantly filling the queue, the more slow the crawlers are, the more I guess the URLs with deeper links are less visited, the access speed is also relatively slow. I caught 90% in the first night, and only captured 95% in the last three days. If I couldn't wait, I stopped it. A total of 5.31 GB images are captured and there are more than 0.3 million files. It should be relatively large, with 1.27 GB of logs. After reading this, it should all be the site of the Institute of deep research. The crawling process has come to an end, and the next step is to calculate PageRank.