My DevOps hands-on experience sharing one or two

Objective

With the system growing, developers, sites, servers more and more, micro-service propulsion, ... And so on, it's increasingly necessary to automate devops.
The real reason, of course, is that these problems were foreseen at the start of the team, so it's up to you to automate from the beginning.
The tangible benefits are also obvious, savings in human costs, standardized processes, traceable release histories, free hands (repetitive labor), avoidance of errors caused by human actions, and so on.

Feel a little bit

1. The current set of products on the market is either expensive or does not support local deployment, or it is a demo level thing, most importantly, each company or product has its own special environment or business. Not necessarily all fit. Anyway is more difficult to find good, and is a complete set of products. Look forward to a devops sap, but also cheaper!
2. Several Big Brother products still do a great job! Like Jira,confluence,jenkins,sonar. The official documentation is very complete and there are many online tutorials. The interface is complete. Unlike some products, it looks tall, and one uses a variety of pits.
3. Understand the development of operation and maintenance is a crazy programmer! (^_^)
4. People are really important, otherwise the flow of something, hehe, is a fart ...

About the way it is.

Of course, there are still a lot of imperfections in this set of tools, which can be further changed with the constant use or changing needs.

Gitlab

Open source Git repositories, mainly for several purposes
1. Source Code Control
Branch management rules can refer to Gitflow, or specify a suitable for their own good, microservices, a site or a project to participate in the developer only a limited number of people. In a simple way, master acts as a release branch, each iteration is developed using a new branch, which is merged into master before it is online, and the simple fix on the line is submitted directly on the master branch.
2. configuration file Management
Placed on the Gitlab, mainly for the convenience of management, as well as retrospective revision history. Of course, we have their own configuration center, can walk the configuration center as far as possible to configure the center, only must be the file configuration management to put on the Gitlab.
3. Release Script management
The publishing script that Jenkins needs to use. Varies by environment, source code language, deployment method, etc.

Jira

Jira Agile Development Management tools, management requirements, research and development iterations. In addition to their company's wiki to do the Knowledge base management is basically stable.
Jira use down to find is still quite powerful! Various customizations are configurable. pages, fields, processes, and so on are all configurable. There are HTTP open APIs that can invoke modify information, trigger process, etc. directly

The publishing process used is also relatively straightforward. Development creates a release task, then submits it to the test, tests the operation on the Jira to the test environment, the alignment environment, the online environment for testing, and so on. After the environment test to publish to the online need to have leader permission to do a review, on the one hand is let leader know what things on-line, on the other hand is also a number of reasons for security control (for example, Holiday Eve is best not to do updates, etc., to do updates must be reported, Otherwise, the holiday will have to belch fart).

is a relatively historical version, recently found in Jira a progress bar plug-in, and then the build published real-time progress directly to the Jira page. This way, you no longer have to open the publishing system to see the progress of the publication and further improve the experience.

Publish process workflows, designed according to your own circumstances

Publishing system

This piece is a simple system that we have developed ourselves. The main role is to connect the use of various open source tools. The various sub-tools that are dispersed as an adhesive system can be linked as a whole.
Although, Jira inside has Jenkins plug-ins, Jenkins inside has Jira plug-ins, but the components of each system have version requirements, and then the components are also quite inconvenient, and finally there are some needs to solve a lot of trouble, so have their own release system.
function or relatively simple, a front-end guy to get the page what's 1 weeks to finish, the key is to put all the new technologies in the current show a wave, although the page is very ugly. Several system interfaces call their own research and write a few days to finish.
Several features that are mostly done
1. Release Configuration Management
Site or system development language, deployment of the target system, to deploy those hosts, is not Docker container way, Docker to deploy a few examples, deployment method concurrent, serial release, to go to that Nginx, bound domain name, bound port and so on information.
When the new system or site released by the Transport and Research and development coordination completed, and later by operations to maintain, such as expansion capacity, etc.

2. Receive Jira release task action notification, and notify to a certain Jenkins to execute, sonar for static code inspection, etc.
3. Receive Jenkins build deployment feedback over the progress
4. Show build Deployment Progress

5. Part of the CMDB system functions, host management (IP, name, user name, password) and the like. Convenient.
As for why it is not necessary to use the existing CMDB system, it is also a rogue, either money, or trouble, or no interface. Soch himself to do one simple. can satisfy the function.
Because it involves the host's account password and so on, so the password is public key encryption stored on the system.
And the password of the use of 2, one is Jenkins in the deployment of the time when the new machine to create SSH login to use once, there is the remote management tool to use, so the use of the password to write a separate small component with the private key decryption to obtain the password, and then the release system and the widget separately managed

Jenkins

Jenkins can be said to be the big guy in this set of tools, it can be said that everything is around him in the turn.
Receive build requests from the publishing system, pull code, compile, pull configuration files, package into deployment packages, upload FTP, publish to private Docker repositories, deploy, and more.
Also differentiate the system environment, the development language (Windows, Linux, Nodejs,. NET core) is handled separately, and so on.
1. Parameterization of the build process. such as the name of the branch to be built.
2. Source code configuration. Git source code address, Gitlab fixed code read-only account, through SSH code pull.
3. Call the build script. The execution commands within Jenkins are approximately as follows

#!/bin/bash -lcd /opt/deployscript # 进入构建脚本目录git pull #拉取最新的构建脚本#调用构建脚本#workspace,build_number,jobname,project_name,git_commit,git_branch,env,jira_id,useridsh /opt/deployscript/${JOB_NAME}/${env}/deploy.sh "${WORKSPACE}" "${BUILD_NUMBER}" "${JOB_NAME}" "${PROJECT_NAME}" "${GIT_COMMIT}" "${selected_branch}" "${env}" "${JIRA_ISSUE_ID}" "${BUILD_USER_ID}"

Jenkins's Build script

The weight is heavy, all the drivers are in this script. A build or deployment script that is written separately for the environment and sub-development languages.
Why each site has a script for the reason is that there are always some sites are so special and excellent, of course, most of the system can walk a common build script.
There are a lot of scripts to call other system interfaces, I write a console application directly with. NET core, which is specifically responsible for this, after all, the shell is not professional.
The specific build script will not be posted.
Script Execution Steps (NET core test environment script): The progress is fed back to the publishing system when each deployment is completed or when an error occurs.
1. The source code in the Jenkins configuration has been helped to pull the good. So the script doesn't have to pull the code.
2. Compile. such as dotnet publish-c release-r linux-x64-o "Output path"
3. Compiling the output content package
4. Upload to FTP.
5. Pull the configuration file.
6. Input content and configuration files, etc. into a compressed package
6. Pull the deployment configuration. To deploy to those machines, deploy to concurrency or serial, etc.
7. Check whether the machine has completed SSH-free configuration, no configuration is configured to pull the password.
8. Publish operations in parallel or serially
9.SSH to target machine, upload compressed package, deploy script
10. Execute the deployment script (unzip, stop the original service, start a new service, check whether the startup is successful, etc.)

Sonar Static code Check

After receiving the Jira's publish request in the publishing system, pull the configuration of the site and send the request to the sonar Jenkins if a sonar check is required.
Currently we configure the release to the production line time to do sonar static code check, and then the sonar system inside the configuration.
Back to see if you want to mail the results of sonar. intend to do so. Weekly out of a code quality report, statistics on the items that have been launched in the week and the previous week compared to errors, loopholes, bad taste, coverage and other data changes. Get a timed mission, sonar. 2 interfaces get the data, store the comparison results, send an e-mail is finished.

Briefly summarize

Article casually write, a lot of things are not clear, there are many things have not said. For example, Fortress machine integration, log, host monitoring integration and so on. I will not say that I am too lazy, typing good tired ah!
In short, welcome to communicate!! Although the implementation is not complete, but it is suitable for the current needs of its own. The right one is the best.

Thanks to the big guy in the open source industry, although I don't have the money to donate. Let the community have so many so many useful products.
Thank the ancestors have planted a good tree, very cool!

The complete set of tools, if the real time estimate is less than one months, of course, the real situation is fragmented, the East stamp, the West poke poke. Fortunately, before doing this thing has a simple plan, no detours, although the road to find domestic products took some time

Starting from the beginning of use, 3 months less than 2000 times, this is still at the beginning of the stage. It is conceivable that productivity tools are indeed