Rafter's DevOps

Boot phase

Over the past 6 years, I have had a unique opportunity to see how our company has evolved from a few fresh graduates who just want to rent textbooks into a big, mature company. When I look back, I will divide our growth into two different stages: before a round of financing and a round of financing, unlike most startups you hear or read, we've been through a long period of a round of financing (about 3 years).

At this stage of development, we do not spend a lot of resources or manpower on the DevOps, instead we focus on product construction. For most of the first few years, I've positioned myself as a software engineer, spending only a few days a month on system management.

We used to joke that I was the one who was "stuck" in the server management. But, in fact, I enjoy myself particularly. Still, I never thought that my enthusiasm for software development would be used in server management.

Try to be as simple as possible (within the range you can control)

There are several reasons why we can not spend our energy on DevOps at the beginning. First, in large part because we are small companies, the changes are relatively few. We chose a simple architecture, and the number of applications is relatively small. So while our products are evolving, there are fewer places to make changes to the underlying infrastructure.

Excessive investment

Second, we've invested too much in server hardware in the early days. We may only need two servers to run our entire site, but we have bought 10 units. But that allows us to spend less time worrying about performance or infrastructure development, and we're only a few years away from the performance of the servers we purchased early. Of course, there are upfront costs for configuring these servers, but once they are set up, there is no need for big changes.

Choose a good tool

Finally, we use Ruby on Rails as an application framework and its associated tools, such as Git, Capistrano, and teamcity, which allow us to use those good release and deployment practices early on. At the same time, we are building on proven and stable open source solutions, such as Nginx, MySQL, and memcached.

It is also because of the adoption of these tools and frameworks that allows us to avoid the adoption of unnecessary complex and proprietary solutions. A lot of times I think these programs will grow with the company and reduce its development speed.

Growth

As our company enters the second stage of growth, our engineers and product teams are growing steadily. The days of only two developers are gone, and in the blink of an eye, we have 10 or even 20 people working on existing and new products. It is conceivable that the number of changes we need to introduce will increase steadily. The absolute number of applications that we need to load on our hardware also goes straight up, first twice times before, then three times times. Developers also want to use new application frameworks, programming languages, databases, queuing systems, and caching servers.

As these complexities increase, so does the cost of errors and downtime. Soon we found that the old manual configuration infrastructure was no longer applicable, and the lack of maturity and flexibility in our infrastructure layer would bring us the following problems: Slowing the release of new products and features and compromising our stability. With that in mind, I stopped working on the product and focused on the research and development automation, monitoring, and publishing process.

Introduction of Chef

Fortunately, we recognize that we need to practise devops. At that time, we encountered the Opscode Chef, and it advocated the "Infrastructure code" principle. Initially, we spent several months writing automated scripts for each part of the existing infrastructure. Once completed, we can use these automated scripts to refactor all servers, which will greatly reduce the burden on our shoulders. Now all of our servers have been set up, and we finally have a place where everyone in the team can see how each infrastructure is built and set up. Equally important, it allows us to quickly add additional resources.

The establishment of DevOps team

DevOps is beginning to assume a unique responsibility within the company, providing key support for our product line and ensuring that our infrastructure continues to expand. In addition to these priorities, we are constantly developing tools and products to manage these infrastructures that require continuous support and improvement. Especially in the early stages, if new applications are introduced into the infrastructure, the underlying automation scripts are required to be modified. As more and more people are starting to rely on our work, internal users (developers and operational personnel) have come up with more devops-related requirements.

As our development team has been divided into different product teams, and each team is focused on different businesses, we have decided to set up a separate DevOps team. This way we have a dedicated team that solves the infrastructure problem full-time. In addition, the availability and stability of application platforms are also important to our business, and downtime or other problems can have a significant impact on our services. We need to ensure that specialized, well-trained engineers are available to assist and investigate issues, especially if the problem is attributable to multiple teams or cannot be understood at that time.

Automatic fruit

On-Demand configuration

After adopting chef and automating the infrastructure of our products, what we do immediately is to improve our testing and staging systems to match them to the same degree of automation. One of the complaints we often hear is that developers cannot quickly and easily show business people what they are doing. To that end, we developed a 100% self-service portal so that anyone in the company could start a pre-configured server to run the full program on EC2.