AWS-based time series processing application architecture

AWS-based time series processing application architecture

When data is used as a periodic continuous measurement, it is called time series information. AWS's unique positioning addresses the scale-up challenges brought by Time Series-based information processing.

This elasticity is achieved through AutoScaling groups for collection and processing, AWS data pipelines for Amazon Elastic MapReduce scheduled jobs, AWS data pipelines for inter-system data arrangement, and AmazonRedshift for large-scale data analysis. Key Architecture elements include SQS for message buffering, which reduces frequent AWS Data Pipeline scheduling and maintains cost prediction and control for the overall solution.

1. remote devices, such as electric meters, mobile clients, advertising network clients, industrial instruments, satellites, and environmental measurement tools, perceive the world around them and use HTTP (S) send sample sensor data for processing.

2. Information sent to the Amazon Simple QueueService queue is further stored to Amazon DynamoDB through the auto-scaling AmazonEC2. A table in DynamoDB is a time-based and week-oriented table structure.

3. If one supervisory control and data collection (SCADA) system exists, you can create a sample data stream from Amazon DynamoDB to support another cloud computing or other existing systems.

4. A scheduled Amazon ElasticMapReduce job is used to create a data pipeline. Intensive sampling processing can be calculated at the same time and sample results can be output.

5. The data pipeline stores the results to Amazon Redshift for further analysis.

6. The data pipeline exports week-oriented historical sampling data tables from AmazonDynamoDB to Amazon Simple Storage Service (Amazon S3 ).

7. The data pipeline exports the results to other acceptable custom formats.

8. Amazon Redshift is used as an option to retain historical sampling data and computing results.

9. With Business Intelligence solutions from internal or Amazon partners, AmazonRedshift can support other large-scale analysis.

Web application hosting architecture in AWS

Architecture of batch processing tasks in AWS

This article permanently updates the link address: