If you are new to DevOps, check out these 5 steps to build your first DevOps pipeline.
DevOps has become the default way to solve slow, isolated or other failures in the software development process. But when you are new to DevOps and unsure where to start, it doesn't make much sense. This article explores what a
DevOps pipeline is and provides 5 steps to create it. Although this tutorial is not comprehensive, it can lay the foundation for you to get started and expand in the future. First, insert a short story.
My DevOps journey
I used to work in Citigroup’s cloud group to develop (Infrastructure as a Service) infrastructure as a service web application to manage Citi’s cloud infrastructure, but I often research on how to make the development pipeline more efficient and bring a positive culture to the team Interested. I found the answer in the book recommended by Greg Lavender. Greg Lavender is the CTO of Citi's cloud architecture and infrastructure engineering (the Phoenix project). Although this book explains the
principles of DevOps, it reads like a novel.
A table at the back of the book shows how often different companies deploy in the release environment:
But before Citi implemented DevOps, this was not the case. Back then, my team had environments with different build stages, but the deployment on the development server was very manual. All developers can only access a development environment server based on the IBM WebSphere Application Community Edition. The problem is that when multiple users try to deploy at the same time, the server will be down, so developers have to notify each other when deploying, which is quite painful. In addition, there are problems with low code test coverage, cumbersome manual deployment process, and inability to track code deployment based on defined tasks or user needs.
I realized I had to do something, and I also found a colleague who felt the same. We decided to work together to build an initial DevOps pipeline - he set up a virtual machine and a Tomcat server, and I set up Jenkins, integrated Atlassian Jira, BitBucket and code coverage testing. This amateur project is very successful: we have nearly fully automated the development pipeline and achieved almost 100% uptime on the development server. We can track and improve code coverage testing, and Git branches can be associated with deployment tasks and jira tasks. together. In addition, most of the tools used to build DevOps are open source.
Now I realize how primitive our DevOps pipeline is, because we did not take advantage of advanced settings like Jenkins files or Ansible. However, this simple process works well, perhaps because of the Pareto principle (also known as the 80/20 rule).
Brief introduction to DevOps and CI/CD pipeline
If you ask some people, "
What is DevOps?", you might get some different answers. DevOps, like agile, has evolved to cover many different disciplines, but most people will at least agree with these: DevOps is a software development practice or a software development lifecycle (SDLC), and its core Principles are a cultural change-developers and non-developers breathe the same sky, things that were previously manual are automated; everyone is doing what they are good at; deployments at the same time become more frequent; Increased throughput; increased flexibility.
Although having the right software tools is not the only thing needed to implement a DevOps environment, some tools are necessary. The most critical one is
continuous integration and continuous deployment (CI/CD). In the pipeline environment, there are different construction stages (for example: DEV, INT, TST, QA, UAT, STG, PROD), manual work can be automated, developers can achieve high-quality code, flexible and large-scale deployment.
This article describes a five-step method for building a DevOps pipeline, as shown in the figure below, using open source tools.
Complete DevOps pipeline
Step 1: CI/CD framework
The first thing you need is a CI/CD tool, Jenkins, which is an open source CI/CD tool under the Java-based MIT license. It is a tool to promote the DevOps movement and has become a de facto standard.
So, what is Jenkins? Imagine it is a magical universal remote control, can deal with many different servers and tools, and can arrange them together. On its own, CI/CD tools like Jenkins are useless by themselves, but become very powerful as different tools and servers are connected.
Jenkins is just one of many open source CI/CD tools for building DevOps pipelines.
Here is what DevOps looks like when using CI/CD tools.
CI/CD tool
Your CI/CD tool is running on the local host, but you can't do anything else yet. Let us follow in the footsteps of the DevOps journey.
The second step: source code control management
The best (and probably the simplest) way to verify that the CI/CD tool can perform some magic is to integrate with the source code control management (SCM) tool. Why do you need source code control? Suppose you are developing an application. Whenever you build an application, whether you are using Java, Python, C++, Go, Ruby, JavaScript, or any language, you are programming. The program code you write is called source code. At the beginning, especially when you are working alone, it may be okay to put everything in a local folder. But when the project becomes huge and invites others to collaborate, you need a way to avoid merge conflicts when the shared code is modified. You also need a way to restore a previous version—the way to back up, copy, and paste is outdated. You (and your team) want a better solution.
This is why SCM has become indispensable. SCM tools help version control and multi-person collaboration by saving code in the warehouse.
Although there are many SCM tools here, Git is the most appropriate. I highly recommend Git, but there are still other open source tools if you like it.
Source control management
CI/CD tools can automate the source code check-in and check-out and complete the collaboration between members. its not bad, right? But how can it be turned into a working application that will be used and appreciated by billions of people?
Step 3: Automated build tools
Awesome! Now you can check out the code and submit the changes to source control, and you can invite your friends to collaborate on source control. But so far you haven't built an application. To make it a
web application, it must be compiled and packaged into a deployable package or executable program (note that interpreted programming languages like JavaScript or PHP do not need to be compiled).
This led to an automated build tool. No matter which build tool you decide to use, they all have a common goal: build the source code into a desired format, and automate the tasks of cleaning, compiling, testing, and deploying to a certain location. The build tools will vary depending on your programming language, but here are some commonly used open source tools worth considering.
awesome! Now you can put the configuration file of the automated build tool into the source code control management system and let your CI/CD tool build it.
Build automation tool
Everything is so beautiful, right? But where is it deployed?
Step 4: Web application server
So far, you have an executable or deployable package file. For any truly useful application, it must provide some kind of service or interface, so you need a container to publish your application.
For web applications, the web application server is the container. The application server provides an environment where the programming logic in the deployable package can be detected, presents an interface, and provides web services to the outside world by opening sockets. In other environments, you also need an HTTP server (such as a virtual machine) to install the service application. Now, I assume that you will learn these things yourself (although I will discuss containers below).
There are many open source web application servers.
Now the DevOps pipeline is almost ready to use, well done!
Web application server
Although you can stop here for further integration, code quality is a very important thing for application developers.
Step 5: Code coverage testing
Implementing code test pieces may be another troublesome requirement, but developers need to catch all errors in the program as early as possible and improve the code quality to ensure end user satisfaction. Fortunately, there are many open source tools to test your code and make suggestions for improving quality. Even better, most CI/CD tools can integrate these tools and
automate the testing process.
Code testing is divided into two parts: "Code Testing Framework" helps write and run tests, and "Code Quality Improvement Tool" helps improve the quality of code.
Code testing framework
Code quality improvement tools
Note that most of the tools and frameworks mentioned earlier are written for Java, Python, and JavaScript, because C++ and C# are proprietary programming languages (although GCC is open source).
Now that you have applied the code coverage testing tool, your DevOps pipeline should be as shown in the figure at the beginning of the tutorial.
Optional step
container
As I said before, you can publish your application on a virtual machine (VM) or server, but containers are a better solution.
What is a container? The brief introduction is that the VM needs to occupy a lot of resources of the operating system, which increases the size of the application, and the container only needs some libraries and configuration to run the application. Obviously, VMs still have important uses, but containers are a more lightweight solution for publishing applications (including application servers).
Although there are other options for containers, Docker and Kubernetes are more extensive.
For more information, check out other articles on Docker and Kubernetes on Opensource.com:
What is
Docker?
Introduction to Docker
What is Kubernetes?
Kubernetes practice from scratch
Middleware automation tools
Most of our DevOps pipelines focus on collaboratively building and deploying applications, but you can also use the DevOps tool to accomplish many other things. One of them is to use it to implement Infrastructure as Code (IaC) tools, which are also well-known middleware automation tools. These tools help complete the automated installation, management, and other tasks of middleware. For example, automation tools can pull down applications with the correct configuration, such as web servers, databases, and monitoring tools, and deploy them to application servers.
Here are several open source middleware automation tools worth considering:
For more middleware automation tools, check out other articles on Opensource.com:
Ansible Quick Start Guide
Ansible automated deployment strategy
Configuration Management Tool Top 5
Future development
This is just the tip of the iceberg of a complete DevOps pipeline. Start with CI/CD tools and explore other things that can be automated to make your team's work easier. And, looking for open source communication tools can help your team work better together.
