DevOps @ Spotify

This article is part of the "DevOps Monthly Combat Story" series. Every month we listen to devops stories from different teams, find out what works for us and which are not, and describe the challenges that DevOps is facing in the implementation process.

Apply DevOps experience to management

There are a lot of blog posts, articles and tweets on DevOps on the Internet. Some discuss their good and bad, while others discuss the implications of introducing DevOps, while others discuss how to implement them.

Although this article mentions some devops aspects of commonality, its main focus is to apply the principles of DevOps to another field: Engineering leadership.

I will use the word "we" from time to time to avoid confusion by making a quick introduction here:

Ingrid Franck: Agile Coach of the engineer team

Ramon van Alteren: Product owner of the engineer team (product manager)

Mattias Jansson (i.e. me): Department head of the engineering team, the team leader mentioned in this article

As of this paper, the Spotify engineer team involved in the article has nine members.

The DevOps culture of Spotify

At the beginning of Spotify's creation, many aspects of devops culture were rapidly pervasive in the sector.

Spotif hired 6 employees in the early stages of entrepreneurship, all of them engineers, one of whom also served as an operator. At the time, the Spotify team had just set up in an apartment, and these people were supporting the entire company. In this context, the initial team of Spotify's understanding of the importance of business profoundly influenced the relationship between transport and peacekeeping development.

After a long journey, Spotify now develops to hundreds of engineers in four cities in three time zones across the country. Although DevOps's mind cannot penetrate the heart and soul of every engineer in the engineer's team, we don't actually mention it anywhere, but you'll find it everywhere in your day to day work, even when you have a little chat time in your tea break.

Startups are mostly composed of developers and business freaks. In these companies, developers usually write code, need to be deployed to maintain the code, before the recruitment of an operation to support engineers. The two camps of Spotify development and operation are highly coincident, and they are very similar in terms of responsibilities, skills and interests. We have a number of operational capabilities of the developers, there are many operational engineers have a strong development background. The great advantage of having these excellent engineers is to solve the day after day, when the problem is still early in the embryonic phase of the removal period.

Back-end development engineers have deployed their code in a production environment, not all of which require the help of engineers to complete deployment. When a backend developer is given such a privilege, the developer takes a more serious look at the tasks traditionally carried out by operation, including monitoring, logging, package management, and usability.

Because our production environment server already has thousands of scale, the Operation dimension engineer more time already does not have to concern the stand-alone machine, but is the cluster. The engineer avoids a manual deployment on a single server, but, as much as possible, with the support of our Configuration Management System (puppet), to apply changes to the online server by modifying the data state of the backend module.

Back-end services usually have the so-called two system owners: one is development (DEV), the other is operational dimension (OPS). Their core responsibilities are in the areas they specialize in. The development engineer is in charge of the code, the design, and the architecture; The operational engineers are in charge of the service-they deploy the service, introduce traffic, and inject life into the service. However, the two systems have great overlap, so the owners of the two systems-development and operation-will periodically discuss the scalability of the architecture, the changes in the background topology, how upcoming new products will affect service behavior, and so on.

We have a team dedicated to automating tools and services, and sometimes it takes weeks for development and operation to work together to solve specific problems that OPS has proposed and prioritized to address.

Despite all these things, our team still has a long way to go. The number of customers, servers, data centers, services, offices, and staff is growing, so the previous solution doesn't necessarily fit the current development, and past solutions constrain the need for scalability today. On this basis, the change of Dev's ratio to OPS staff has diluted DevOps's mentality in some degree.

Learned from DevOps.

DevOps has taught us a lot, but the most important thing is to teach us how to coordinate development and operational goals. Let them work together to provide them with the space to learn from each other. Through regular communication between the two groups, developers will have the opportunity to understand and think about why it is sometimes necessary to play the role of an impediment (stumbling), to learn how to plan ahead to adjust the requirements of the operating environment when the production environment changes. Meanwhile, once operational dimensions begin to realize that their hardware and services are not static, but rather a data structure that can be reshaped by code, developers will also treat the system in a different way: he cares no longer just the code in the package, but whether the process of deploying the package to the production environment is flexible enough.

Similarly, injecting this operational thinking into the development process can effectively reduce the frequency at which operational engineers are interrupted and cleaned up, ensuring that they have more time to invest in long-term projects.

In general, DevOps's work has changed the way the team develops and transports the views of the entire system, as well as the recognition of values, not just in the traditional sense of responsibility for their respective positions.