OpenStack deployment and O & M practices

OpenStack deployment and O & M practices
Introduction to OpenStack

OpenStack is an open-source IaaS implementation. It consists of several associated sub-projects, including computing, storage, and networks. Since its launch as an Apache protocol, more than 2010 companies have joined the OpenStack project since its establishment in 200, including AT&T, AMD, Cisco, Dell, IBM, Intel, and Red Hat. Currently, more than 17,000 developers are involved in the OpenStack project, from 139 countries. This figure is still growing.

OpenStack is compatible with some AWS APIs. To provide more powerful functions, it also provides OpenStack APIs (RESTFul APIs ). Compared with other open-source IaaS, the architecture is loosely coupled, highly scalable, distributed, pure Python implementation, and friendly and active communities make it very popular, the Development Summit once every six months also attracted developers, suppliers and customers from all over the world.

The main sub-projects of OpenStack include:

• Compute (Nova) is the core of OpenStack and is responsible for managing and creating virtual machines. It is designed for easy scalability, supports a variety of Virtualization Technologies, and can be deployed on standard hardware.
• Object Storage (Swift) is a distributed, scalable, and multi-copy Storage system.
• Block Storage (Cinder) provides Block Storage services to provide persistent Block-level Storage devices for OpenStack virtual machines. Supports multiple storage backends, including Ceph and EMC.
• Networking (Neutron) provides network virtualization services. It is a pluggable, scalable, and API-driven service.
• Dashboard provides a graphical console service that allows users to conveniently access, use, and maintain resources in OpenStack.
• Image (glance) is an Image service designed to discover, register, and deliver Virtual Machine disks and images. Multiple backend services are supported.
• Telemetry (Ceilometer) provides the Usage Statistics Service, which allows you to conveniently implement the OpenStack billing function.
• Orchestration (Heat) integrates many components in OpenStack, similar to AWS's CloudFormation, allowing users to manage resources through templates.
• Database (Trove) is a database-as-a-service built based on OpenStack.

Netease private cloud uses four components: Nova, Glance, Keystone, and Neutron.

Overview of Netease private cloud platform 1. NetEase private cloud architecture

The private cloud platform of Netease is developed by Netease Hangzhou Research Institute. It mainly provides infrastructure resources, data storage and processing, application development and deployment, O & M management, and other functions to meet the company's product testing/launch needs.

Figure 1 shows the overall architecture of Netease private cloud platform. The entire private cloud platform can be divided into three categories of services: core infrastructure services (IaaS), basic platform services (PaaS), and O & M Management Support Services, which currently include: cloud hosts (virtual machines) cloud network, cloud hard disk, Object Storage, object cache, relational database, distributed database, full-text retrieval, message queue, video transcoding, Server Load balancer, container engine, cloud billing, cloud monitoring, management Platform and other 15 services. The private cloud platform of Netease makes full use of the latest results of cloud computing open source. We have developed and deployed VM instances and cloud network services based on the keystone, glance, nova, AND neutron components of the OpenStack community.

Install and deploy Openstack on Ubuntu 12.10

Ubuntu 12.04 OpenStack Swift single-node deployment Manual

OpenStack cloud computing quick start tutorial

Deploying OpenStack for enterprises: what should be done and what should not be done

CentOS 6.5 x64bit quick OpenStack Installation

To deeply integrate with other services (such as cloud disks, cloudmonitor, and cloud billing) on the private cloud platform of Netease And meet the specific requirements of the company's product usage and O & M management, based on the Community OpenStack version, our team independently developed the following features: cloud host resource quality assurance (computing, storage, and network QoS) more than 20 new features, such as image block storage, heartbeat reporting of VM instances, and Intranet isolation of tenants in flat-dhcp mode. At the same time, our team summarized some deployment, O & M specifications and upgrade experience in daily O & M of OpenStack and the new Community version. Over the past two years, the OpenStack team of Netease's private cloud platform has been adhering to the open-source and open-source concepts and has always followed the principle of "Source Community and feedback community. While enjoying the continuous R & D of new functions and bug fixing in the OpenStack community for free, our team also actively contributes to the community to help the OpenStack community grow and grow. Over the past two years, our team has submitted nearly 100 commits for new function development/bug fixes to the community, and fixed over 50 community bugs, these community contributions involve the Essex, Folsom, Havana, Icehouse, and Juno versions of OpenStack.

Thanks to the increasingly stable and mature OpenStack, the private cloud platform has been running stably for more than two years, providing services for Netease's 30 Internet and game products. From the perspective of application results, the Netease private cloud platform developed based on OpenStack has achieved the following goals:

1. This increases the company's infrastructure resource utilization and reduces hardware costs. Taking the CPU usage of physical servers as an example, the private cloud platform increases the average CPU usage from 10% to 50%.
2. Improves the level of infrastructure resource management and O & M automation, thus reducing O & M costs. With the help of self-help Web resource application and allocation methods and automatic deployment services on the cloud platform, the system O & M staff reduced by 50%.
3. The elastic usage of infrastructure resources is improved, and the adaptability to product business fluctuations is enhanced. Using virtualization technology, physical infrastructure is made into a virtual resource pool. With effective capacity planning and on-demand use, the private cloud platform can well adapt to sudden business spikes.

Introduction To Netease OpenStack deployment reference solution

In a specific production environment, to ensure both performance and reliability, the keystone backend uses Mysql to store user information and memcache to store tokens. To reduce access pressure on keystone, The keystoneclient of all services (nova, glance, neutron) is configured to use memcache as the token cache.

Because Netease private cloud needs to be deployed in multiple data centers, each data center is naturally isolated geographically, which is a natural disaster tolerance method for upper-layer applications. In addition, to meet the functions and O & M needs of the private cloud, Netease private cloud needs to support two network modes at the same time: nova-network and neutron. To address these requirements, we have proposed an enterprise-level multi-regional deployment solution, as shown in figure 2. On the whole, the deployment between multiple regions is relatively independent, but the Intranet can be used for intercommunication. Each region includes a complete OpenStack deployment, therefore, you can use an independent image service and an independent network mode. For example, region A uses the nova-network, and Region B uses the neutron, which does not affect each other. In addition, to achieve single-point login for users, keystone is shared among regions, which are divided based on the network mode and geographical location.

Figure 2. multi-region deployment Method

Unlike the typical OpenStack deployment, hardware is divided into computing nodes and control nodes. In order to make full use of hardware resources, we strive to design the deployment as symmetric, if a node is deprecated, the overall service is not affected. Therefore, we divide the hardware into two categories: computing nodes and control computing nodes. Compute nodes are deployed on the following nodes: nova-network, nova-compute, nova-api-metadata, and nova-api-OS-compute. In addition to compute node services, the control compute Node also deploys nova-schedauth, nova-novncproxy, nova-consoleauth, glance-api, glance-registry, and keystone, as shown in 3.

APIS provided externally include nova-API-OS-compute, nova-novncproxy, glance-api, and keystone. This type of service is stateless and can be easily scaled horizontally. Therefore, all such services are deployed after the Server Load balancer HAProxy and Keepalived is used for high availability. To ensure service quality and ease of maintenance, we did not use the nova-api, but managed separately by the nova-api-OS-compute and nova-api-metadata. In terms of external dependencies, Netease private cloud has deployed high-availability RabbitMQ clusters, master-slave MySQL clusters, and memcache clusters.

Figure 3. computing node Control

In terms of network planning, Netease private cloud mainly uses the FlatDHCPManager + multi-host network mode of the nova-network and divides multiple VLANs, they are used for virtual machine fixed-ip networks, Intranet floating IP networks, and Internet networks respectively.

In O & M, the self-developed O & M platform of Netease is used for monitoring and alarm. The functions are similar to those of Nagios, but they are more powerful. Important monitoring alarms include log monitoring and process monitoring. Log Monitoring ensures that the service is detected immediately when an exception occurs, and Process Monitoring ensures the normal operation of the service. In addition, Netease private cloud uses Puppet for automatic deployment and StackTach to help locate bugs.

For more details, please continue to read the highlights on the next page:

1 2 3 Next Page