Breakthrough applications from cloud storage

In recent years, with the rapid growth of mass video information, traditional security technology is more and more difficult to meet the needs of some industries in transmission, storage and large data calculation and analysis, or it is difficult to meet these needs with lower cost, more flexible extensibility, more robust, more reliable and more secure system architecture. Therefore, the concept of cloud computing in the two years began more and more infiltration into the field of security. The breakthrough application of cloud storage is particularly noticeable. This article will mainly on the application of cloud storage in the field of security analysis.

The popularization of network technology, making the network sharing of video resources become more and more urgent demand of many industries, the existing distributed video resources are moving towards the direction of large concentration; Safe city, Intelligent transportation and other major industries market through the point, line, face of intensive coverage, every moment are adding new video sources; 720p,1080p, 4 K, higher video resolution means greater video information ...

These growing video data bring a huge amount of video storage requirements, but also to the current video storage technology put forward a series of new challenges: How to achieve the smooth expansion of storage capacity? How to effectively reduce the resulting storage costs? How can you more reliably save, use more securely, and share this data more easily? How to solve the problem of maintenance management caused by increasingly large system?

The answer lies in the cloud storage.

The definition of cloud storage

Cloud storage is a technology or service that derives and develops in the cloud computing concept. The collection of various types of storage devices in the network in the cloud computing system, through the concept of cloud computing to organize work together, the external provision of data storage and business access functions and storage-related value-added applications, known as cloud storage.

Cloud storage and cloud computing are the technical implementations of the cloud architecture, and any application requires a computing model, a storage model, and a communication model. In cloud computing, the reuse of resources must be realized in order to realize the flexible utilization of resources and to create the illusion of unlimited resources for the users. The same is true of cloud storage, which maximizes the benefits of resources by virtualizing and streamlining configuration technologies.

Compared with traditional storage devices, cloud storage is not only a hardware, but a complex system consisting of a network device, storage device, server, application software, public access interface, access network and client program. Each part takes the storage device as the core, through the application software to provide the data storage and the business access service externally.

Cloud storage provides storage-related services externally. Cloud storage can provide a bare space service similar to a disk array, or it can provide a specific service based on storage.

System Architecture for cloud storage

To understand the system architecture of cloud storage, you need to understand its application architecture first. We take Ping An city as an example to see the application architecture of cloud storage. In a safe city, cloud storage can provide storage services for multiple systems, including video surveillance, card-port Alarm, image resource base, and analysis of graph and reconnaissance. It can complete the daily video recording resources, the various cases involved in the video and pictures and other resources, the card electric police generated by the vehicle capture pictures and other resources of the unified storage.

From the perspective of application architecture, the difference between cloud storage system and traditional distributed storage system is not obvious, but the more essential difference is embodied in the internal software architecture.

Cloud storage System with large-scale distributed parallel file system, based on a large number of servers and storage equipment, build a large-scale storage cluster, providing hundreds of PB of storage capacity, and can be online capacity expansion, the overall cost of building a large-capacity storage system is far lower than the traditional storage architecture, and has good extensibility and flexibility.

The asymmetric architecture of the cloud storage system, separated by metadata and storage data, through load balancing and data concurrency access strategy, under ordinary hardware conditions to obtain up to dozens of Gbps transmission rate and the hundreds of petabytes of storage capacity, and according to user application development trend, timely online dynamic expansion. Unlike the stand-alone file system, the Distributed file system does not place the data on a single disk, which is managed by the upper operating system, but resides on a server cluster, where the servers in the cluster work together to provide services to the entire file system.

The cloud storage System has built-in object-based data management policies, it can ensure the security and reliability of the data in the system, and completely eliminate the single point of failure in the storage system, and combine the automatic fault detection and fast recovery technology to ensure the user's application keep running stably, and reduce the difficulty of deployment and management.

More intuitive understanding, the cloud storage system itself is built on a universal disk array, which manages data on disk through the operating system's APIs, except that such a system can logically be divided into metadata nodes (control nodes), data nodes (storage nodes), management nodes, and client four parts, These four sections correspond to the 4-layer structure model of the cloud storage mentioned above:

1 Metadata node (Control node)

The metadata node, which is the control node, is used to record various attributes of the stored files, which is equivalent to the brain of the whole file system, manage each data node, collect data node information, understand the status of all data nodes, then assign tasks to them, coordinate and instruct each data node to serve the system;

2 Data node (storage node)

Data node is a storage node, used to manage disk and volume and store user's file data, is the storage resource provider of the whole storage system, it is also responsible for cluster construction, including node management and data slicing, and undertakes data redundancy protection and load balance of data access;

3 Management Node

Management node Unified Management cloud Storage System cluster nodes, files, logs, alarms, gateways, while providing log management, performance monitoring, configuration management, fault management capabilities, to facilitate system management and maintenance personnel Management cloud storage system;

4) Client

Clients in an independent process to provide services, to provide users with file reading, directory operations and other APIs, when users need to use the Distributed File system for file reading and writing, the client installed to the server to use the system, you can use the services provided by the system.

Three metadata nodes are used in Koda storage System, which plays an important role in protecting metadata (redundancy) and load balancing (improving system performance). The data node periodically reports its status to the metadata node, waits for and processes the command, realizes the data efficiently and safely reads and writes. In the system, the size of the data slice can be adjusted to meet the requirements of different customers on the data file Storage policy, and the full mirror copy is used to protect the data files.

Comparative advantages of cloud storage and traditional storage systems

Cloud storage has significant advantages over traditional storage in the following areas:

1 flexible expansion of storage capacity

The maximum capacity of traditional storage equipment is limited, it is difficult to expand when it reaches a certain scale, while storage performance, security, reliability and economy are greatly reduced. Cloud storage can easily obtain PB-level storage capacity through clustering, storage expansion is unlimited, can add storage nodes to meet storage capacity requirements, anytime, anywhere, and be able to "increase storage nodes, automatic data reorganization, reduce storage nodes, automatic data recovery" flexibility and scalability.

2 high concurrent read and write performance

The concurrent read-write performance of traditional storage devices is easily restricted by the ability of the NAS head, CPU or controller, while the cloud storage system adopts the architecture of control and data separation, which is separated from the ability of single equipment, and the performance of the whole system is limited by the network throughput ability. Cloud storage technology can aggregate the bandwidth of storage nodes, and the bandwidth can be linearly increased with the increase of storage nodes, theoretically the bandwidth is infinite. At the same time, in the cloud storage system, data files are split into blocks of data to be stored on more than one physical storage node, which can access data concurrently. In addition, the data storage in cloud storage is stored in the multi copy strategy, and can realize the load balanced access of hotspot data.

3 maintainability

The cloud storage system uses the data redundancy storage mechanism, when the hard disk or the storage node is damaged, the remaining nodes can be reorganized automatically, the data is not lost, the system operation is not affected. This greatly improves the maintainability of mass storage nodes. At present, the most intuitive and most able to embody the advantages of cloud storage is maintainability. Take the Koda storage system as an example: traditional storage devices, usually in a RAID mode for redundant backups, when there is a hard disk damage, the raid refactoring time is usually more than 10 hours, if the hard drive is broken, the data can not be restored, so the maintenance staff need to be in the first time to replace the hard drive, 7x 24-hour standby; Koda storage uses a distributed file system, data storage and backup no longer rely on the ability of a single device, when the hard drive is broken, the rest of the hard drive will be automatically refactored, restore the damaged data, this will only take more than 10 minutes, the maintenance staff only need to check the damage of the hard disk regularly, and replace the new hard drive, Maintenance is very simple.

4) Security, reliability

Traditional storage devices provide a transparent storage space, the original data directly on the storage device, data can be directly accessed and used, does not have information security and privacy. Data transfer in cloud storage is encrypted, and first the user does not know which physical hard disk the data exists on, and the data is stored on the storage device by file block and cannot be accessed directly. The data stored by the user can only be accessed and managed by itself, and the system administrator cannot read it.

5) resource Sharing

Traditional deposit? Devices typically work in a single form, unable to aggregate capacity and bandwidth between multiple devices. and cloud storage cluster file system, not limited by the number of hardware storage nodes, can be flexibly unified management and sharing. Cloud storage can share a single storage pool with multiple users, and traditional storage is limited by capacity and bandwidth performance. Cloud storage provides a globally unified namespace, provides an accurate file-access interface, supports mainstream file transfer protocols, and supports API interfaces for better integration of applications to achieve optimal access efficiency.

6 Total cost of ownership is low

The cloud storage System employs a cost-effective Ethernet network and storage nodes, which has a better cost-performance advantage over traditional high-end storage devices. Cloud storage capacity expansion is relatively flexible, according to the business anytime and anywhere flexible expansion, users do not need to buy in the upfront, when needed, online performance and capacity expansion, so as to protect the user's upfront investment. More importantly, cloud storage can greatly reduce the total cost of ownership (TCO) of the user in management maintenance through the aforementioned maintainability.

Concluding

The storage characteristic of video image is high demand for real-time access performance, and the system architecture of cloud storage determines that it can satisfy the concurrent writing and reading of mass HD or SD video with low total cost of ownership, and can realize fast configuration and Plug and play.

Therefore, although in the short term, the demand of cloud storage will focus on the main market of Safe city, intelligent transportation, and civil market based on public cloud structure, but it is believed that the future will be more expansive with the more comprehensive popularization of network and the rapid growth of mass video sharing demand.