Cloud computing Overview

Original translated from: http://www.cnblogs.com/loofywang/archive/2013/01/16/MyUnderstandingOfCloudComputing.html

Cloud computing covers distributed computing, parallel computing and grid computing, distributed storage, parallel processing, and grid sharing.

Cloud computing is a group of servers used to save and process data.

The business model of cloud computing is to lease storage space and data processing services to make profits for users. The advantage of cloud computing is that the cost is low and resources can be shared in a centralized manner. Cloud computing stores data with greater emphasis on data reliability. data redundancy is generally added to ensure data reliability, for example, multiple data backups.

The cloud computing model is a producer-consumer model. The grid computing model is a resource sharing model. You can see the differences from the following figure:

1. Cloud Computing Architecture

2. Grid Computing Structure

 

Common cloud computing data storage technologies include GFS developed by Google (Google File System) and HDFS developed by hadoop (hadoop Distributed File System.

1. GFS: Manages large-scale distributed data-intensive computing and Scalable Distributed File Systems. It uses low-cost commercial hardware to build a system and provides fault-tolerant and high-performance services. It differs from a common distributed file system,

GFS is composed of a master and a large number of block servers. The master stores metadata, and files in GFS are cut into 64 MB blocks for storage. GFS ensures data reliability through data redundancy. Each data is backed up in at least three copies. To ensure data consistency, all backups are modified when the data is modified, the version number is used to maintain the consistency of the backup. The read operation is not performed on the master. When reading data, it only obtains the location of the target data block from the master and then reads the data from the block server. Write operations first obtain authorization on the master, and then distribute the data to be written to all data backups. After each data backup is obtained, the system sends a signal indicating that the master backup has been obtained, after all data backups obtain the written data, the primary data backup informs the user end of this signal. The user end sends the write signal to the primary backup, and the primary backup sends the signal to all backups, after each data backup is written, a signal is sent to the master backup. After all the data backups are written, the master backup sends a signal to the user end to complete the data writing.

The Data Management Technology of cloud computing is famous for its bigtable developed by Google.

1. bigtable: a distributed storage system that manages structured data. It can manage petabytes of data on thousands of Taiwanese servers. It optimizes read operations and uses column storage. Its data storage format is <row: String, column: String, time: int64>-> string, the basic elements are rows, columns, timestamps, record boards, and record boards. Data items in bigtable are arranged in the Lexicographic Order of rows. Each row is dynamically divided into different record boards. Each node manages about 100 record boards. The timestamp is a 64-bit integer, indicates the version of the data item. Bigtable has three main components during execution, which are linked to the client database, master server, and record board server. The primary server is used to allocate a record board to the record board server for load balancing and garbage collection. The record board server is used to manage the record board and process read/write requests. Bigtable uses three layers to store data location information. The first layer is chubby file, which stores the location information of the root tablet. The second layer is the root tablet, which stores the location information of the metadata tablet, the third layer is the metadata tablet, which contains the location information of many user tables, and the user table is the place where the specific data is stored.

Most programming models of cloud computing are based on map-Reduce. The so-called map-Reduce refers to dividing a large task into multiple small tasks, which are then processed in parallel by multiple servers, finally, sum up the results to produce the final result. It is a programming model of Parallel Computing for task scheduling and execution. It mainly implements two functions: map function and reduce function. map function defines the method for processing segmented data. Reduce function defines the induction and integration of intermediate results after map function processing. Map-ReduceProgramData is automatically divided into blocks, map jobs and reduce jobs are assigned to various work processes and scheduling each work process. Exceptions and errors are also automatically handled. The workflow of Map-Reduce is as follows:

1. The map-Reduce database divides the input data into N parts, copies user program processes into M parts, and puts them into different servers. One of the copied processes is the master, the other is worker. The master is used for scheduling and management of worker. For example, a map or reduce job is assigned to an idle worker. Map and reduce jobs are implemented by the user. You can configure the number of data blocks and the number of process copies.

2. The worker process of the assigned map job reads the corresponding input data block. Each input data block has a key-value pair. After the map job finishes processing the block data, the intermediate result is saved to a new key-value pair. The new key-value pair is first cached in the memory and regularly written to the local disk, these new key-value pairs are placed in different zones, and each zone has a worker process that executes the reduce job. The location information of these new key-value pairs is reported to the master, and the master sends the location information to the worker process that executes the reduce job. The worker that executes the reduce job sorts these key-value pairs first and then performs integration, and outputs the results to a result file. Multiple output files are generated. You can use these files as input, call another map-Reduce program, or merge them.

Note that map and reduce Jobs call map and reduce functions multiple times to process data.

The map-Reduce program has

1. When a worker error occurs, block the worker and move the preceding program to another worker for execution. This process is controlled by the master.

2. When a master error occurs, replace the master according to the checkpoint.

Currently, the three major cloud computing technologies are implemented by Google and hadoop. Map-Reduce is implemented by Google using C ++ and hadoop using Java. Google is GFS + bigtable + map-Reduce, hadoop is HDFS + hbase + map-reduce.

Problems that cloud computing can solve:

1. Fast big data processing and analysis

2. lower the threshold for ordinary users to use high-performance computing capabilities, that is, they can provide low-cost high-performance computing capabilities.

3. allows common users to access massive data and process it as needed

Existing cloud computing problems:

1. Cloud computing provider interfaces are inconsistent, and there is no uniform interface standard. As a result, service providers cannot uniformly deploy services in different cloud computing providers to avoid DDoS (Distributed Denial of Service) attacks.

2. Security

3. Data Transmission bottleneck

The shared nothing architecture is a distributed computing architecture in which each node is independent and independent from other nodes, without single point of competition. Bigtable and map-reduce both adopt this architecture.