Open-source news collector (I) Content Overview

1. I have nothing to worry about recently. I want to sort out my work recently.

Goal: News collector

1. You only need to enter the list URL. The Collector will automatically collect all the articles.

2. the collector does not need to write any collection rules at last.

3. html paging acquisition policy based on static crawlers (self-made, inaccurate)

4. Open-Source project-based content extraction algorithm (html2article)

5. Code-based open-source probe project (nuniversalchardet)