High-availability Data acquisition platform (how to play the 3-door language Php+.net+aauto)

High-availability Data acquisition platform (how to play the 3-door language Php+.net+aauto)

Similar articles: architecture applications for high concurrency data acquisition (Redis applications)

Spit Trough: I am the main course is PHP, the team is not proficient in. NET talent, in order to solve this platform solution, or a part of the effort.

New Year, hope to have a fresh start. Technology + team management all have a new breakthrough, in the new year to oneself better, can not continue to base.

Point of departure:

´ with the increasingly powerful software, user demand for more and more, users also hope that a large number of data integration, to achieve the rational use of resources.

´ Some data resources need to crawl the form of Web pages to collect data.

´ Collection application is not unified, no good management procedures, disorganized.

´ collection applications are often unresponsive and have no corresponding interrupt procedures, which are greatly applied to the instability.

Solution:

In order to solve the above problems, the following solutions are proposed, mainly using the concept of service platform.

Data Acquisition Service : Create a Windows service using C #. (Aauto has not found an instance of creating a Windows service to date)

Features include:

1. Periodically request pending collection and put in the queue to be collected

2. Multi-threaded timing uploads resource data and other status information through the API.

3. Timed platform monitoring, acquisition application time too long to send an interrupt notification, notification unresponsive force to close the app and restart.

4. Upload heartbeat information to ensure the normal operation of data collection service.

Acquisition Application: use Aauto to create an application, in fact, using. NET can also create a collection of applications, mainly considering the Aauto language this team is relatively mature point, and already have a ready-made application, just a little modification can be connected with the service platform.

Features include: 1. Collect data resources into local databases, such as SQLite.

2. Save time-consuming information, through the time-consuming computing service platform can decide how long to issue the interruption notification, and can predict how long to complete the work.

status applications: mainly auxiliary applications.

Features include:

1) can view processing progress

2) can view data upload progress

3) can view pending progress

4) can view application time-consuming information

through the above Solutions : Service platform as an integrated worker, responsible for interacting with the PHP API, relatively independent of the collection application, so there is a good decoupling, but also to ensure stability.

The collection application is no longer responsible for interacting with the PHP API, which is primarily responsible for interacting with local resources, so the amount of code is reduced and the acquisition speed can be improved accordingly.

Specific code implementation: You can pay attention to the next post, I will also enter the amount of code to contribute. Feel good to give a praise, New year atmosphere.

