Restructure The R&D Project to Promote Automated O&M DevOps

In the article, I incorporated a lot of my own ideas. Of course, more than half of the landing plans for our company's DevOps have been implemented, and I'm still working on the blueprint for the rest.

background
Let’s talk about the background first. We are a small company. Although we are under the banner of making products, each customer has a large number of personalized functions. This refers to the java, Android, and ios ends of each customer (most of the function codes are Same, different personalized function codes). I used to do Android before, and practice has proved that under special circumstances, only our Android group can switch to any customer and any version of the code. And fix a public bug, all customers' versions will be updated. I have also been introducing this development model, but I did not get support. Until the beginning of the year, I was promoted to the mobile team leader. Plus, after the java team leader changed jobs, I had the opportunity to fully implement the refactoring plan.

Foreword
Because I have done many years of operation and maintenance (Internet cafe software and hardware operation and maintenance, etc.), I have a high understanding of server hardware and software, plus I have some development experience in various languages. It is a full-stack engineer. Both are familiar. All this has brought great help to my implementation.
I affirm that all of the following restructuring ideas are derived from my own ideas. For implementation, I will arrange the training after unified implementation. (Although the implementation resistance is great, but the final effect is good)

Let me talk about the effect first
Before, it took 2 weeks for us to complete a java project, Android project, ios project and deployment server environment for each customer project. Now, we need about 30 minutes. And no matter how much each customer has personalized needs, we can also flexibly develop and switch to each customer's code.
Before, our development process was extremely chaotic. There were no documents and various development processes. Now we are gradually standardizing and saving at least 50% of development costs. Of course, we are constantly improving.

java project team
Our java project was originally divided into interface services and background management services, but they are all in a git repository. I divided it into:
Interface service
Background Services
html5 package
Public package service
Because of this refactoring, I gradually adopted the front-end separation scheme, so there are more h5 packages.

Each library is divided into multiple branches, where master is defined as the main branch, and each customer is a newly opened branch. Through branches to solve the various needs of each customer, (because the customer needs are too detailed, some texts have to be changed, simple In the case of plug-in development, each plug-in needs more than n versions, so that we can’t do it for our small company), of course, each customer should also have a development branch, but it is limited by our small staff, and the first phase is omitted. Development branch, the local temporary storage area is used as the development branch.
For how to develop in different branches, please refer to my other article
"Are you sure you can remember so many git commands? Try Sourcetree quickly"

The development process is as follows:
Developers only need to debug locally and submit the code to a project branch of the git library, which is automatically compiled by Jenkins. For Jenkins automatic compilation of full-end projects, please refer to another article.
A small company continuous integration tool jenkins practice (JAVA WEB, Android, IOS, html)
If the compilation error is sent back to the developer's mailbox that affects the code via email, in addition, after the tester deploys it with one click, if the test has a problem, it can also be sent to the developer through the jira bill of lading. After the developer receives it, continue to submit the code. No longer like us before, the developer must be notified, the developer packs locally, so the project code cannot be tracked.

supplement:
Because we have many projects, it takes time and effort to maintain the version number manually. I decided to use Jenkins to automatically fill in the version number into the project and reflect it in the file name. Therefore, the package compiled by the project may be 1.war, 2 .war, 3.war, we internally (1, 2, 3) as the version number, of course, I also wrote the version number inside the war package.
Secondly, we removed a lot of state-containing code, so that each war automatically loads different configurations to run in the test environment and production environment. Because of statelessness, the war package and h5 package are both a set of code in the test environment and production environment. Here we spent a few weeks to complete the extraction of stateless codes.

One-click deployment:
We have made a set of management platform that can upgrade all wars and h5 under tomcat middleware. For related introduction, please see another article
Java web project war package automatic upgrade deployment plan
You can see our current effect:

The main functions are as follows:
Upgrade, display current version
The validity period of the war package, the authenticity check
Developers put public sql into public .sql
The customized SQL of each customer is put into each customer.sql
You only need to click the upgrade button at the back to upgrade the sql script or service.
The article was written before and there are some logical changes, so I won't introduce them here.

Client group
The client development process, here ios and Android say that after we submit the code, Jenkins will produce two sets of ipa and apk with different connection server addresses at the same time. Because there are many situations in different network environments, you need to look at the test environment and the production environment, so , I adopt the plan of sending 2 packages at the same time.

In the process of listing the appstore on iOS, I adopted a semi-automated package delivery strategy, uploaded it to the developer platform, and manually listed it.
For each customer and each version, we adopt a multi-branch solution, similar to the above java, different branches handle different customer projects.

Test group
Test group process

After the tester got the bill of lading from jira, the developer will see the specific version number after the solution, and then enter the oss storage, install the Android program or ios program, and deploy the java project with one click.
And because Android and iOS both send two packages at the same time, the java project and h5 project are stateless, so both the production environment and the test environment can be quickly deployed and tested.
In the figure, the end of the real file name is the address of the connection to the production environment.

project manager
Project manager process
I adopt the new process of new customers, the old process of old customers, and gradually migrate to the new process.
The execution of the database script has been omitted, and it is executed by the unified deployment platform.

Operation and Maintenance Group
This picture is the logic of the past, because each period, the server and server system given by each customer are different, and the operation and maintenance are extremely complicated. I use the unified docker container solution to solve this problem, making each client machine environment the same.

We integrated the servers of each customer into a docker cluster, and then managed by our company's unified management platform, and then assigned each container to different roles. Here we used Portainer to solve it and developed some requirements for the second time.
The management here does not require the password of the Linux machine, which is controlled by the TLS certificate, and the Windows machine will be abandoned.

Advantages of this solution:
Under docker, all customer production environments are the same.
Easy to backup, migrate and restore.
A high availability environment can be built, and grayscale is used when sending packages, and blue and green deployments do not interrupt service.
It can support elastic scaling design and expansion.
Support load balancing, domain name forwarding, accidental container switching, etc.
It is conducive to transforming microservice architecture.
Quickly build an environment
The establishment of a docker cluster requires us to complete automation. Here I use the Ansible tool to implement it. We can use instruction distribution to obtain the version of a certain package on all machines and execute different codes.

Then use image mirroring to deploy containers to customers uniformly. Here we have established a private server.

Private Service
Here I have established various warehouses to facilitate java development. I have established a private warehouse, an official maven agent warehouse, and an Alibaba Cloud agent warehouse;
In order to facilitate the development and packaging of other environments on docker, I created a docker private warehouse;
There are also some npm warehouses created to solve the unified management of linux servers;
And the gradle private warehouse to be used by Android;

Private server
Big data

Big data has always been the core product of many companies. For small companies, how to implement it at low cost? I have studied a set of powerful big data frameworks and made some secondary developments.
Rest assured, as long as you use this framework, you can easily solve various big data chart problems.

Interface
I use the restful style API to develop the interface in an all-round way, and the interface documentation is automatically generated. There are several articles involved here. I haven't had time to write it yet.

Specification
I have also formulated some specifications and conventions, such as internal development and cooperation, which have not been written yet, and I will make up later.

Cultural articles
Previously, our development needs were entirely allocated by the project manager. Now I arrange for continuous reconstruction of the project, which brings a lot of new processes. Here, with the consent of the leaders, we arrange to allocate accounts to each department every week or every two weeks. Implementation. This involves the company's internal business, inconvenience, please understand.

The landing of DevOps
The word DevOps was known at the 2017 CNUTCon Global O&M Technology Conference in Shanghai last time. I learned that most companies are doing automated O&M and reducing O&M.
In the article, I incorporated a lot of my own ideas.
Of course, more than half of the landing plans for our company's DevOps have been implemented, and I'm still working on the blueprint for the rest. Will definitely share with you in the future.