[In-Service software engineering] Basic Cloud computing Tutorial

The first basic article

Introduction to the first chapter

Basic concepts of cloud computing:

Narrow-sense cloud computing:

Cloud computing in broad sense:

Basic features of cloud computing:

Iaas:infrastructure as a service

Paas:platform as a service

Saas:software as a service

Benefits of Cloud computing:

Advantages and Disadvantages:

Chapter II Architecture and standardization of cloud computing

The second piece of technology

Chapter III Cloud Storage

The structure model of cloud storage

Two types of architectures:

Block storage, file storage

Fourth Chapter cloud Services

The fifth chapter of virtualization

Sixth Chapter Cloud Desktop

Chapter Seventh Cloud Security

----------------------------------------------------------------------------

Principles and practices of cloud computing (http://book.2cto.com/201308/30081.html)

• 2nd Cloud Computing Core Technology

There are many ways to classify virtualization. From the point of view of virtualization, virtualization is divided into three main categories: Platform virtualization, resource virtualization, and Application Virtualization.

Platform Virtualization (Platform virtualization), which is the virtualization of the entire computing environment and the running platform, is virtualized at the computer and operating system level, including server virtualization and desktop virtualization. Server virtualization can use a single physical server as a number of servers. Desktop Virtualization will decouple the desktop environment from the devices it uses, and the server holds a full desktop environment for each user. Users can log on to the same desktop from anywhere, and within the company, it is easy to manage users ' desktops in a unified way. Server virtualization is achieved primarily by virtualizing the main resources of the server, such as CPU virtualization, memory virtualization, and I/O interface virtualization.

Resource Virtualization (Resource virtualization), which is the virtualization of server resources, mainly includes: Memory virtualization, storage virtualization, network virtualization, and so on. Memory virtualization is the virtualization of the server's memory, which forms a number of independent memory blocks allocated to the virtual machine for use. Storage virtualization is the consolidation and management of the entire cloud system's storage resources, providing users with a unified storage space. Network Virtualization is the addition of an abstraction layer between the underlying physical network and the network user, which splits down the physical network resources and provides a virtual network upward. Depending on the type of network virtualization, network virtualization generally includes virtual LAN and virtual private network, thus realizing the virtualization of the whole network environment.

Application Virtualization (Application Virtualization), which abstracts applications from dependencies on the underlying operating system and hardware, unlocks the coupling between the application and the operating system and hardware. Application Virtualization includes simulation, simulation, and interpretation techniques. There are Microsoft Application Virtualization (App-V), VMware ThinApp, Symantec software Virtualization solution (SVS) on behalf of the product.

Platform virtualization technology can be subdivided into the following major categories:

(1) Full virtualization

Fully virtualized (full virtualization) is the virtual machine that simulates the complete underlying hardware operating environment, including CPU, memory, disk, NIC, etc. through virtualization. Some of the protected privileged instructions between the guest operating system and the original hardware are captured and processed by the hypervisor (hypervisor), which allows the client operating system to run without modification, as shown in structure 2-1.

Full virtualization runs faster than hardware simulations, but the performance is not as bare-metal as hypervisor needs to occupy some resources. The biggest advantage of full virtualization is that the operating system has not undergone any modifications. Its only limitation is that the operating system must be able to support the underlying hardware. Representative products include VirtualBox, KVM (kernel-based Virtual machine), VMware Workstation, VMware ESX Server, and Virtual pc.

(2) Semi-virtualized

Semi-virtualized (paravirtualization) technology is also called hyper-virtualization, which, like full virtualization, accesses the underlying hardware through a hypervisor. The difference is that semi-virtualization, while fully simulating the entire underlying hardware, needs to modify the code of the client operating system section to access the privileged state, so that it can interact directly with the virtual Machine Monitor (MONITOR,VMM).

Compared to full virtualization, para-virtualization does not require the hypervisor to compile and capture privileged instructions, and therefore runs faster. The disadvantage is that the client operating system needs to be modified and users are more difficult to use. Represents a product with Xen, Hyper-V, etc.

(3) Part of virtualization

Partial virtualization (partial virtualization) means that hypervisor only simulates some of the underlying hardware, so the guest operating system is not modified to run in the virtual machine, and other programs may need to be modified.

(4) Hardware-assisted virtualization

Hardware-assisted virtualization (Hardware assisted virtualization) refers to the help of hardware (primarily the host processor) for efficient full virtualization, with the goal still full virtualization. Create virtual machines with hardware help to monitor and allow the guest operating system to run independently.

In full virtualization and semi-virtualized, privileged instructions need to be processed by software, and hardware is much faster than software, so you can use hardware to pull a lot of virtualization logic out of the software, greatly simplifying the virtualized product architecture. Hardware-assisted virtualization technology is often not used alone, but with full virtualization and semi-virtualized virtual optimization, play an auxiliary role. Many of the fully virtualized and semi-virtualized products available today support hardware-assisted virtualization, such as VirtualBox, kernel-based virtual machines (KVM), VMware Workstation, VMware ESX Server, Xen, Vt-x, Amd-v, and more.

(5) OS-level virtualization

Operating System level Virtualization (virtualization) is a lightweight virtualization technology used in server operating systems that isolates different processes by creating multiple virtual operating system instances (cores and libraries). Processes in different instances do not understand each other's existence at all.
With operating system-level virtualization, all virtual servers must be running the same operating system (although each instance has its own application and user account), with less flexibility, but operating system-level virtualization requires little modification, low cost, and good running performance

At present, more is server virtualization, to realize server virtualization must realize the server virtualization of the three major hardware resources. At the same time, in order to better dynamic resource management, dynamic migration of the server must also be realized. The following is a description of the server three hardware resource virtualization and real-time migration technology.

1. CPU Virtualization

CPU virtualization is the abstraction of a single physical CPU to simulate multiple CPUs in parallel, allowing one server to run multiple operating systems at the same time, multiple operating systems are independent of each other, and the application can operate in a separate operating system environment without affecting each other. At any one time, only one virtual CPU instruction can be run within a single physical CPU. The purpose of CPU virtualization is to improve CPU performance and utilization. Virtual CPUs run independently of each other and interfere with each other. Multiple virtual machines provide services at the same time, keeping the CPU busy for a long period, greatly improving CPU utilization.

CPU virtualization in the x86 architecture can be implemented with full virtualization, semi-virtualized, or hardware-assisted virtualization. Full virtualization is primarily based on the use of binary code translation techniques and privileged-level compression techniques. Binary code translation is the process of scanning and modifying the client's binary code to translate the hard-to-virtualize instructions into virtualization-enabled instructions when the virtual machine is running. The privileged-level compression technique is to have the client and hypervisor run under different privilege levels. Because of the different privilege levels, the hypervisor can intercept and virtualize a subset of the privileged instructions executed on the client. Semi-virtualization differs from full virtualization by using privileged-level compression, which modifies the client operating system code to translate the privileged instruction-related operations into a super-call (Hypercall) of the hypervisor (Hypervisor). Full virtualization and semi-virtualized are all software-only CPU virtualization, the x86 processor itself is not modified, the middle cost is large. Hardware-assisted virtualization enables the CPU to join the new instruction set and run mode to allow the hypervisor and the guest operating system to run in the appropriate mode, respectively, to complete the functions associated with CPU virtualization. Both Intel and AMD have introduced hardware-assisted virtualization products, such as Intel VT Technology and AMD-V technology, which are both CPU hardware-assisted virtualization technologies.

2. Memory Virtualization

Memory virtualization is to give the physical memory to the hypervisor unified management, usually using a block to wrap it into multiple virtual physical memory allocated to several virtual machines to use, so that each virtual machine has its own independent memory space, non-interference. The hypervisor maintains a mapping relationship between the physical address of the client to the host's physical address. The goal of memory virtualization is to isolate the virtual machine memory space, and the idea of block sharing makes the entire virtual machine think that it has the entire memory address.

Memory virtualization under the x86 architecture can also be implemented with full virtualization, semi-virtualized, or hardware-assisted virtualization. Full virtualization using the Shadow Page table (Shadow page) technology, the Shadow page table technology is the client operating system memory page table records the memory logical address to the client operating system to see the "physical" address mapping relationship, while the hypervisor maintains a corresponding page table, Log the machine memory address mapping relationship of the "physical" address that the client operating system sees to the real physical machine. The page table maintained by the hypervisor changes as the page table of the client operating system changes, so it is called the Shadow page table. When there is memory access, the hypervisor gives the shadow page table to the Memory management Unit (Management Unit,mmu) for address translation and obtains the true physical address of the memory. The use of Shadow page table technology on behalf of the product has KVM, VMware Workstation, VMware ESX Server. Semi-virtualized uses page table write technology, which is required to register the page table with the hypervisor when the client operating system creates a page table. Then, when the client runs, the hypervisor deprives the client operating system of write access and continuously manages and maintains the page table. When a program on the client operating system accesses memory, it can obtain a real machine address directly in its own page table. Xen is a representation of page table write technology. Hardware-assisted virtualization uses the Extended page table (Extended page table,ept) technology, which expands page table technology by using hardware to add a page table to the original page table, and the added page table is called an extended page table. With this page table, programs on the client operating system can directly translate the client's memory address into a real machine address when accessing memory, thereby mitigating the overhead of full memory virtualization.

3. I/O interface virtualization

The I/O interface virtualization is the unified management of the physical machine's real devices, packaged into multiple virtual devices to be used by several virtual machines, responding to device requests and I/O requests for each virtual machine. I/O interface virtualization includes virtual network cards and virtual switches. The goal of I/O interface virtualization is not only to give virtual functions accurate and fast access to the I/O resources they need, but also to isolate them, avoid confusion, and more importantly, increase the utilization of I/O resources while mitigating the overhead associated with virtualization.

Virtualization of I/O interfaces in the x86 architecture can also be implemented with full virtualization, semi-virtualized, or hardware-assisted virtualization. Full virtualization is achieved by fully simulating I/O devices. Semi-virtualized is a front-end/back-end simulation, and Xen is using this approach. Hardware-assisted virtualization uses direct partitioning to directly assign physical devices to a client operating system, directly accessing I/O devices by the guest operating system (not through hypervisor). Currently, the main hardware-assisted virtualization I/O technologies are Intel's vt-d, AMD Iommu, Pci-sig Iov (I/O virtualization), and so on.

4. Real-time migration technology

Live Migration (live migration) technology refers to the full and fast migration of virtual machines from the source host hardware platform to the new host while ensuring that the services on the virtual machine are running properly. The mobile process users will not be aware that the virtual machine requires only a very short period of downtime.

The live migration process requires hypervisor assistance, which requires that the hypervisor on both the source and destination hosts interact with each other to complete replication of the virtual machine's running state. Before replication begins, the destination host cleans up resources to ensure that there is sufficient resource space to receive the virtual machine. After replication begins, the memory pages are continuously copied from the source host to the destination host, and the replication process does not affect the operation of the source host. When the replication is complete, the destination virtual machine runs, the source virtual machine terminates, and the live migration process is complete.

Live migration can be used to eliminate hazards, resource maintenance, and resource optimization. When a virtual machine resource is tight, it can be migrated to another host with more resources in real time, and the migration process does not affect the operation of the virtual machine. When a virtual machine needs maintenance, the virtual machine can be temporarily migrated to another host, maintenance can be migrated back, the entire process users feel no change. Migrating virtual machines from a resource-intensive host to a resource-rich host facilitates virtual machine consolidation and resource optimization.

The two real-time migration technologies currently used are VMware's vmotion technology and Xen's live migration technology. Xen Live Migration Technology uses pre-replication migrations, which significantly reduce the virtual machine downtime required for the migration process compared to simple stop replication.

Several modes of 2.3.2 data storage

The massive data storage is the hotspot and the difficulty of the computer storage research nowadays. Massive data storage is faced with problems such as difficult to manage, not easy to expand, less efficient to access, and security. Therefore, the ideal mode for mass data storage must be able to access high efficiency, scalability, cross-platform, high reliability, high security and other data sharing capabilities.

Data storage patterns are shown in cluster 2-3.

Data storage is divided into closed-system storage and open systems based on the server type. Closed system mainly refers to the mainframe, not open to the outside, their own independent operation. The open system generally refers to a small workstation that includes Windows, Linux, Mac OS and other operating systems, and the different work sites are connected together through the network to achieve complex computing. Open System storage is divided into: built-in storage and plug-in storage. Built-in storage refers to the storage that is cured on a hardware device. Plug-in storage refers to the storage that is connected to the server via the bus. The open system's plug-in storage is divided into the following ways: direct-attached storage (direct-attached Storage,das) and networked storage (fabric-attached Storage,fas). Direct-attached storage is often connected via a SCSI connection and networked storage via a network cable. Networked storage is divided into: network-attached storage (network-attached Storage,nas) and storage area networks (Storage areas Network,san), depending on the transport protocol.

NAS is an additional device that uses a storage device as a storage system and is connected to the system through a network. Network-attached storage is a dedicated file server that is less computationally-intensive than a normal server, so it is fully functional. It is a technology that integrates distributed, independent data into a large, centrally managed data center to facilitate access to different host and application servers. NAS is a file-level networked storage model, as shown in structure 2-4. NAS has its own file system, which generally provides file access services through the network File System (SYSTEM,NFS) or the Universal Internet File system (Common Internet Files system,cifs). NAS provides a unified storage interface, and all storage devices are connected to the existing network with the same network topology, so the NAS is well-shared. NAS is more than just a storage device, it is more of a device for data backup and recovery.

A SAN is a fast, dedicated subnet that connects storage devices such as disk arrays, tapes, and related servers through network-connected devices such as fibre switches, fiber routers, and fiber hubs. A SAN is a block-level networked storage model, as shown in structure 2-5. Storage area networks are divided into Fibre Channel (Fiber CHANNEL,FC) sans and IP sans, depending on how they are connected. FC Sans, which use Fibre Channel as the transmission medium, overcomes the traditional cable limitations associated with small computer system interfaces (Small computer system INTERFACE,SCSI), greatly extending the distance between servers and storage, thus increasing the likelihood of more connections. However, the transmission distance of the FC San is typically less than 50 km. Even so, for the local area network, generally enough. IP San Technology is a SAN storage technology that architectures a SAN storage network on traditional IP Ethernet and connects servers and storage devices via IP Ethernet. The IP SAN technology uses centralized storage, which greatly improves the utilization of storage space.

The advantages of NAS are very good cross-platform, data sharing on different platforms is very convenient, but the NAS server is located in the middle of the client and storage devices, on the data transmission link, when the number of requests for services surges, the central NAS server processing capacity will become the I/O bottleneck of the entire storage system, The bottleneck of storage performance is the reason why NAS failed to dominate storage mode. In contrast, sans have the advantages of high capacity, high reliability, fast transmission, and the same San also has a disadvantage of poor cross-platform, because there is no unified storage System interface, data sharing is inconvenient. Given the advantages and disadvantages of NAS and SAN, object storage technology combines the benefits of NAS and San two storage architectures with the benefits of shared data for Nas and the fast and direct access of Sans. Object storage technology provides object-based access to objects that are closer to people's natural lives and are easier to accept.

Object Storage (object-based storage,obs) is typically made up of three parts: Client, MDS (Metadata Server), and object storage device (objects Storage device,osd), as shown in structure 2-6. OBS is an object-level storage mode. Client is the customer, is the part of the user direct contact, mainly used to send data operation request, including data reading and writing; MDS is a metadata server, is the basis of object storage system, mainly responsible for metadata management, and ensure the consistency of data access; OSD is the basis of object storage System, is the storage device for object data. Compared to a typical storage device disk, the OSD is similar to a small computer with its own processor, memory, storage disk, network interface, etc., the main function is to manage local objects (object). The OSD is a collection of object. object is the base unit for the storage of objects. Each object contains two parts: a file's data and a set of data attributes. The property set of the data records the properties of the data, which can be customized by the administrator. The properties of the data display the characteristics of the data and facilitate the operation of data synchronization. In the traditional storage, the block device needs to record the location of each storage data block on the device, which needs to occupy a certain amount of space, management and trouble; the OSD in object storage has some intelligence, object maintains its own attributes, greatly simplifies the complexity of storage system management, decomposes management tasks, Reduces the management pressure on the storage system while increasing flexibility.

Although object-oriented storage is still in the research stage, and there is no marketization, but it is closer to natural language to describe things, compared to other storage technologies, object storage technology has high performance advantages, storage device intelligence, data sharing easier, more convenient management, better security and so on. Object storage is the future development direction of data storage technology.

[In-Service software engineering] Basic Cloud computing Tutorial