[Reprint] Gradually build enterprise DevOps ability

When the software industry enters the era of the Internet, the market for the delivery of software products and services put forward higher requirements: not only to quickly realize the demand, but also to quickly release on-line, and must ensure that the business reliable, efficient operation. To meet these requirements, it organizations need strong processes, technologies, and people as a guarantee.

ThoughtWorks early recognition of the importance of release and operation for successful delivery . Our founder Roy Singham, in the last kilometer of business software,[1], states:

the "last kilometer" of the so-called "software development" refers to the stage when the software meets the functional requirements and is not yet operational and creates business value. software Developers-especially software developers who face delivery pressures-often turn a blind eye to the "last mile." but it is really becoming the biggest pressure point in the delivery of business software.

This paper will analyze the typical problems of large software organization in software release and operation Maintenance, and introduce an effective solution.

1. Questions
Many large software organizations experience the following two pressures during the release, operation and maintenance of software:

(1). Quick response
The traditional concept of a large scale, the release cycle of a few months or even years of software product development approach is changing. In the internet age of "Fast fish eating slow fish", time-to-market becomes an important factor to measure the ability of software organization: The software products that can quickly accept the demand, quickly complete the development, and put into use quickly, can occupy the market effectively and attract the users.

With the help of an agile development approach characterized by iterative development and a group of efficient development tools represented by Ruby on Rails, many software organizations have significantly improved their ability to achieve functional requirements. However, from the point of view of business owners, it is not enough to improve the efficiency of the development phase to achieve end-to-end rapid response. Many software organizations are developing iteratively, but release and deployment still take place every few months at the same pace as before. From the perspective of the customer and the end user, the delivery of these software organizations is still done by waterfall: the customer and the end user do not directly perceive the benefits of the development capability Improvement (1).

Without effectively shortening the rollout cycle, you can't really quickly respond to business needs and quickly realize business value. How to shorten the cycle of publishing and operations work has become a problem that has plagued many software organization leaders.

Figure 1: Iterative development + Waterfall release [2]

(2). Quality
Large software organizations often attach great importance to product quality and invest significant cost and effort into quality assurance activities during the development/testing phase. However, the quality of software products is not only introduced in the development stage, but also by the traditional testing work can not be fully discovered. A considerable proportion of the quality issues are introduced or discovered after the dev/test phase. The causes of this phenomenon are:

• Developers lack knowledge of the production environment and introduce flaws in the code that are only exposed in the production environment.
• Developers lack focus on non-functional requirements and do not have a corresponding validation environment, resulting in non-functional defects.
• Production environments and test environments lack effective management because environmental differences introduce defects.
• Lack of automation in deployment and maintenance, and manual introduction of defects during the release process.
• Lack of regression testing for production environments leads to defects not being discovered in time.

Through the introduction of automated testing, test-driven development, continuous integration and other agile practices, the development/testing phase of quality assurance activities can be effectively improved. However, for both the customer and the end user, any defects introduced in the link will also cause losses to the business. How to ensure quality under the urgent pressure of deploying on-line is a problem that many software organization leaders are concerned about.

2. The attempt of agile pull-through

Some software organizations realize the existence of these problems, and hope to take the agile development method as the starting point, the downstream release, deployment, operations and other work links pull-through, thereby improving the overall responsiveness. But due to some inherent differences between software development and operations, such pull-through activities are often fraught with difficulties:

• The development team has different concerns with the operations team. the development team attaches importance to achieving business value with functional requirements, and the operations team values the business value of non-functional requirements (stability, performance, security, etc.).
• The development team has a different skill structure from the operations team. developers often lack the skills of server management, and operators often lack the skills of software programming.
• The development team is different from the tools that the operations team uses daily. configuration Management, Ides, test tools, etc. introduced for the development phase are rarely used by operations teams.
• The development team is different from the daily work environment of the operations team. developers typically work on desktop computers within the company, and operators often work on the customer site and on the server.
• The development team and the operations team usually belong to different departments.

Due to these inherent differences, it is difficult to effectively help the operations team to improve by simply extending the practice of agile software Development to the operations team from the perspective of the development team. In order to effectively improve the quality and efficiency of operation and maintenance work, we need to introduce the process, technology and tools in accordance with the characteristics of operation maintenance.

3. Countermeasures

In view of the challenges faced by modern large-scale software organizations in the process of software release, operation and maintenance, ThoughtWorks recommends building devops[3] capabilities in software organizations, thereby enhancing it integration across the organization and improving the quality and efficiency of the "last mile" of software delivery. Lay the groundwork for business agility.
DevOps is a group of processes, technologies, and tools that promote communication, collaboration, and integration between development, technology operations, and quality assurance departments. The name "DevOps" refers to the seamless integration of development (DEV) and Operations (OP). DevOps-capable organizations are able to develop fast, responsive, and stable business operations that keep pace with the innovation of the development process, enabling the benefits of agile development to be demonstrated at the organizational level.

(1). Lean operations
Traditional software operators tend to avoid modifying functionality as much as possible, thereby reducing the risk of meeting non-functional requirements. However, if a small modification is rejected, and the total amount that needs to be modified within a given period of time is not changed, the size of each change becomes larger, increasing the risk of each release (because the change involves a larger scope).
The guiding ideology of DevOps is "lean operations". Many of the principles of lean production, such as shortening delivery cycles, eliminating waste, valuing value flows, pulling production, and building quality, are all reflected in DevOps. In contrast to traditional software releases,DevOps improves efficiency and quality through changes in the following ways:

• reduce the scope of changes for each release. In contrast to the traditional waterfall development model, the iterative approach means that more frequent releases and fewer changes are included in each release. Because deployments are frequent, each deployment does not have a significant impact on the production system, and the application gradually grows at a smooth rate (2). The large-scale, infrequent release of traditional development methods ( usually in "quarterly" or "yearly" units ) significantly increases the release frequency ( usually in "days" or "weeks" ) compared to organizations with devops capabilities.
• strengthen development and operation coordination. Bridging the skill gap and communication gap between development and operations through a strong release coordination mechanism; Use collaboration tools such as teleconferencing, instant Messaging, Enterprise Portal (wiki, SharePoint) to ensure that all stakeholders understand the content of the change Use unified processes and tools such as story walls, burndown charts, online project management tools (such as Mingle, JIRA), configuration management tools (such as Subversion, Git, Mercurial), and more.
• Automation. reduce the cost of deployment operations, ensure repeatable deployment tasks, and reduce the likelihood of deployment errors with powerful deployment automation and standardized environmental management. For example :
  • Standardize production environments with virtualization technologies such as VMware or Xen, enabling rapid replication and rapid recovery of production environments.
  • Automate environment settings, software Installation/configuration, and more with tools such as Puppet or chef to turn configuration information into source code for versioning of your environment's configuration.
  • Automate the deployment of software products with tools such as Capistrano to achieve version control of the deployment process.
  • Automate database changes with tools such as Dbdeploy to achieve version control of data migrations.
  • use selenium, cucumber and other tools Smoke testing and regression testing in automated production environments.

Figure 2: Application grows at a smooth rate [4]

From the work flow, coordination mechanism, technical tools and other aspects of the simultaneous start, you can build a devops capability in the software organization, so that the lean operations into reality.

(2). Agile development
DevOps and Agile Software development also have a lean guiding ideology, and there are many similarities in the practical aspect. Agile Software development can be regarded as the implementation of lean thinking in demand and development phase, while DevOps is the implementation of lean thinking in the release and Operation phase (3). While building devops capabilities does not necessarily require software organizations to have agile software development capabilities, the following agile practices are particularly helpful for devops capacity building:

Figure 3:devops and agility are both relevant and different [5]

• iterative development. development teams that are accustomed to fixed short-cycle iterations are better able to integrate into the overall rhythm of fast delivery.
• automated testing. an effective automated test suite ensures system quality at every step of the software lifecycle and avoids the introduction of defects.
• continuous integration. with proven project automation mechanisms and capabilities, development teams can help operations teams quickly build automated systems for release and maintenance processes, enabling continuous delivery of software value.

(3). Benefits
By building DevOps capabilities, software organizations can significantly deliver quality and efficiency in the process of releasing and operating software products. In particular, the perceived benefits are:

• Shorten the delivery cycle and new requirements can be put into use faster and create business value.
• Increase the reliability of software release, reduce the quality accident after the line.
• Reduce waste in release and operations and increase operational team productivity.
• Visualize and measure software delivery processes to quickly identify issues and improve continuously.
• Establish a more efficient collaboration between the development and operations teams.

4. Case Studies

Case I:flickr
Flickr is the world's largest picture-sharing site. According to the 2007 statistics [6],Flickr has more than 8.5 million registered users, stores more than 3 billion photos, and responds to 40,000 photo access requests per second .
By automating infrastructure, sharing versioning, automating building and deploying, sharing metrics, and strengthening communication mechanisms, Flickr has been able to deploy more than 10 requests per day , while maintaining the stability and performance of the site. At the same time, the development team and the operational team to establish a mutual respect, mutual trust of the collaborative relationship.

Figure 4: Flickr, the world's largest photo-sharing site, has more than 10 deployments per day [7]

Case II: An online social networking site
The website has been in operation since 2000 and currently has more than 30 million registered users. As the business grows, the site's operations team feels pressured by business leaders and end users. According to ThoughtWorks's value stream analysis, the site will take 15-40 days from the time it takes to accept a demand to finally put it online, with more than 50% wasted.

Figure 5: Discovery of waste through value stream Analysis [8]

ThoughtWorks helped the site to build DevOps capabilities, especially in infrastructure automation, environmental automation, test automation and deployment automation capabilities, and improved the workflow of development and operations teams to reduce the delivery time of typical requirements by more than 50%. Effective working time is more than 90% , which enables the website to achieve full business agility.

5. Challenges

DevOps capacity building is a systematic project, and many factors may affect it. Some of the most common risks are listed below:

• collaborate across departments . Many large software organizations divide development and operations into different departments, and DevOps requires a seamless integration and close collaboration between developers and operations, which inevitably involves coordination between departments. If mishandled, departmental walls can severely damage the ability of software organizations to deliver business value.
• High-level leadership input . DevOps is a change in the way it works compared to traditional waterfall releases, involving changes in technology, processes and even team culture. If there is a lack of high-level leadership attention, or if senior leaders only consider DevOps as a small-scale, technical improvement, DevOps will be hard to get the desired results.
• Team stability . in the traditional sense, "operation and Maintenance" is a low-tech post, and the turnover rate of personnel is relatively high. DevOps requires development teams and operations teams (especially operations teams) to have more comprehensive skills, especially project automation skills. If the team is not guaranteed to be relatively stable, the learning investment will be wasted.

The software release process is a holistic system that requires end-to-end process optimization. ThoughtWorks uses lean value Stream Improvement (Lean Value stream improvement) as a framework for devops construction, where it embeds knowledge and practices for software building, publishing, and operations to iteratively manage improvement activities Visually visualize the progress of your work throughout the process to maximize the success of your improvements.

[1] "Meditation on software development", people's post and Telecommunications publishing house, 2009, chapter II

[2] Image source: Damon Edwards Blog "What Is DevOps" (http://dev2ops.org/blog/2010/2/22/what-is-devops.html, or HTTP/// article.yeeyan.org/view/139515/170072)

[3] Wikipedia's "DevOps" entry: Http://zh.wikipedia.org/wiki/DevOps

[4] Photo source: Wikipedia's "DevOps" entry (HTTP://ZH.WIKIPEDIA.ORG/WIKI/DEVOPS)

[5] Image source: Damon Edwards Blog "What Is DevOps" (http://dev2ops.org/blog/2010/2/22/what-is-devops.html, or HTTP/// article.yeeyan.org/view/139515/170072)

[6] Data source: April MySQL Conf and Expo and Flickr website.

[7] Photo source: Deploys Per Day:dev and Ops cooperation at Flickr (http://www.slideshare.net/jallspaw/10- DEPLOYS-PER-DAY-DEV-AND-OPS-COOPERATION-AT-FLICKR)

[8] Image source: ThoughtWorks Internal data

Thank Propellerhead for the review of this article.

Blog to: Infoq's author Bear festival released on March 26, 2007 "Building DevOps capabilities for business Agility"

Another note: 51CTO on the devops topic: "DevOps is not as simple as you think"

[reprint] Gradually build enterprise DevOps capabilities