Principles and Practices of Atitit web crawlers and data collectors attilax v2 and atititattilax

Principles and Practices of Atitit web crawlers and data collectors attilax v2 and atititattilax

 

Principles and Practices of Atitit web crawlers and data collectors attilax v2

 

1. Data collection 1

1.1. http lib1

1.2. HTML Parser Parsers, 1

1.3. Chapter 1 web crawling 8th 1

2. Implement the class library framework 2

3. problems and difficulties (html to txt) 2

4. References 3

 

 

1. Data collection

Obtains pagesUrls, artUrls, and picUrls.

Refer to locomotive

 

1.1. Http lib1.2. HTML Parser Parsers ,

Chapter 2 web crawling 8th
Author: nickname: old wow's paw (full name: Attilax Akbar Al Rapanui Attila Akba Arla Panui)

Chinese name: atira (iron), EMAIL: 1466519819@qq.com

Reprinted please indicate Source: http://www.cnblogs.com/attilax/

 

8.1 A simple crawler algorithm 199
8.1.1 width priority crawler 201
8.1.2 crawler with preference 201
8.2 achieve topic 202
8.2.1 webpage acquisition 202
8.2.2 web page resolution 202
8.2.3 Delete useless words and extract stem words 204
8.2.4 link extraction and standardization 204
8.2.5 crawler trap 206
8.2.6 webpage library 206
8.2.7 concurrency 207
8.3 General crawler 208
8.3.1 208 scalability
8.3.2 coverage, freshness and importance 209
8.4 crawler limit 210
8.5 topic crawler 212
8.5.1 topic locality and clue 213
8.5.2 optimal priority variation 217
8.5.3 adaptive 219
8.6 evaluation criteria 223
8.7 crawler ethics and conflict 226
8.8 Latest Progress 228
Literature comment 230

 

 

I borrowed books from the library this semester and accidentally saw a book.Network RobotJavaProgramming Guide. I felt like I was lucky.

Crawlers on the market can be said to have no books. Basically, they are only mentioned in search engine books. They only talk about ideas and there is no usable code.

2. Implement the Class Library Framework

Httpclient webdriver

/AtiPlatf_auto/src_atibrow/com/attilax/dataSpider/DoubanSpider. java

 

DoubanSpider be =NewDoubanSpider ();

Be. search (kw );

Be. clickFirst ();

Be. processShowMainTxt ();

 

3. Problem and difficulty (convert html to txt)

Jsoup does not seem to work... Htmlpaser is better ..

It seems that the best way to save as is to use ff in the browser ..

 

Atitit.html conversion extract plain text txt

 

HTML Parser1

Jsoup1

Browser saveas1

4. References

HTML html

 

(1) _ Sina Blog .html

Using Python to write web crawlers ([Australia] Richard ...)

 

Python network data collection (...) [Overview _ book reviews _ online reading]

~ $ Itit webdriver and concept .docx

 

Atiend