What mature framework does java-PHP or python have for data collection and analysis?

I now need to perform automatic data collection on the list of articles on a website and the actual content in the list. the list can obtain the id of each article, each article uses a unified interface (with the article id included in the parameter to obtain the corresponding json... I now need to perform automatic data collection on the list of articles on a website and the actual content in the list. the list can obtain the id of each article, each article uses a unified interface (the parameter can be taken with the article id to obtain the corresponding json), and a part of the data needs to be collected and then analyzed.

Is there any mature framework or wheel that can meet my needs? (Multithreading is required, and it can run stably for 7x24 hours, because the collection volume is huge)

In addition, I would like to ask how to store the collected content (millions to tens of millions). There are some digital data in the data that requires statistical analysis. Can I use mysql? Or are there other more mature and easy-to-use wheels?

Reply content:

I now need to perform automatic data collection on the list of articles on a website and the actual content in the list. the list can obtain the id of each article, each article uses a unified interface (the parameter can be taken with the article id to obtain the corresponding json), and a part of the data needs to be collected and then analyzed.

Is there any mature framework or wheel that can meet my needs? (Multithreading is required, and it can run stably for 7x24 hours, because the collection volume is huge)

In addition, I would like to ask how to store the collected content (millions to tens of millions). There are some digital data in the data that requires statistical analysis. Can I use mysql? Or are there other more mature and easy-to-use wheels?

For data analysis.
Map-reduce for log analysis
Dpark can solve PV and UV analysis
Spark is also good.
After the data report is produced, you can use Pandas for analysis and presentation ..

For data collection. There are a lot of tools.

How do I think you want to build a search engine... Large. Distributed things are recommended.
It is unrealistic to use MYSQL...

Young man, aren't you a crawler?

1. Crawler framework: scrapy

2. Database selection: you can use MySQL to index the database for another five hundred years.

You can also try to use MongoDB

You didn't say any language or environment. For multithreading, nodejs and python are generally used. Both of them can use mysql and other storage data. Tens of millions is not a problem.

Have you ever played python selenium + PhantomJs?

The scrapy in python is still