Principles and Practices of Atitit web crawlers and data collectors attilax v2 and atititattilax

Source: Internet
Author: User
Tags stem words

Principles and Practices of Atitit web crawlers and data collectors attilax v2 and atititattilax

 

Principles and Practices of Atitit web crawlers and data collectors attilax v2

 

1. Data collection 1

1.1. http lib1

1.2. HTML Parser Parsers, 1

1.3. Chapter 1 web crawling 8th 1

2. Implement the class library framework 2

3. problems and difficulties (html to txt) 2

4. References 3

 

 

1. Data collection

Obtains pagesUrls, artUrls, and picUrls.

Refer to locomotive

 

1.1. Http lib1.2. HTML Parser Parsers ,

Chapter 2 web crawling 8th
Author: nickname: old wow's paw (full name: Attilax Akbar Al Rapanui Attila Akba Arla Panui)

Chinese name: atira (iron), EMAIL: 1466519819@qq.com

Reprinted please indicate Source: http://www.cnblogs.com/attilax/

 

8.1 A simple crawler algorithm 199
8.1.1 width priority crawler 201
8.1.2 crawler with preference 201
8.2 achieve topic 202
8.2.1 webpage acquisition 202
8.2.2 web page resolution 202
8.2.3 Delete useless words and extract stem words 204
8.2.4 link extraction and standardization 204
8.2.5 crawler trap 206
8.2.6 webpage library 206
8.2.7 concurrency 207
8.3 General crawler 208
8.3.1 208 scalability
8.3.2 coverage, freshness and importance 209
8.4 crawler limit 210
8.5 topic crawler 212
8.5.1 topic locality and clue 213
8.5.2 optimal priority variation 217
8.5.3 adaptive 219
8.6 evaluation criteria 223
8.7 crawler ethics and conflict 226
8.8 Latest Progress 228
Literature comment 230

 

 

I borrowed books from the library this semester and accidentally saw a book.Network RobotJavaProgramming Guide. I felt like I was lucky.

Crawlers on the market can be said to have no books. Basically, they are only mentioned in search engine books. They only talk about ideas and there is no usable code.

2. Implement the Class Library Framework

Httpclient webdriver

/AtiPlatf_auto/src_atibrow/com/attilax/dataSpider/DoubanSpider. java

 

DoubanSpider be =NewDoubanSpider ();

Be. search (kw );

Be. clickFirst ();

Be. processShowMainTxt ();

 

3. Problem and difficulty (convert html to txt)

Jsoup does not seem to work... Htmlpaser is better ..

It seems that the best way to save as is to use ff in the browser ..

 

Atitit.html conversion extract plain text txt

 

HTML Parser1

Jsoup1

Browser saveas1

4. References

HTML html

 

(1) _ Sina Blog .html

Using Python to write web crawlers ([Australia] Richard ...)

 

Python network data collection (...) [Overview _ book reviews _ online reading]

~ $ Itit webdriver and concept .docx

 

Atiend

 

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.