Jenkins+gitlab Automated compilation deployment scenario exploration and service-side compilation Webpack combat

I. BACKGROUND

Prior to our development process, we compiled the Webpack package locally, and then SVN submitted the source code and the compiled code. Both the source code and the compiled code are updated from SVN before each commit. There are several drawbacks to doing this:

1. SVN updates and commits the compiled code resulting in a large number of conflicting files

2. Since we use a non-covered publishing naming method, it is easy to omit some files after the group has multiple optimizations to submit the test, and when you collate the list of files that need to be published

3. When it comes to multi-person development of the same function, it is easy to generate code is overwritten, manually scheduled release priority, manual comment Others not online code, etc.

4. SVN branch development cumbersome, aggravating workload

The most intolerable is the first 2nd, so we changed to server-side packaging coding, local only to submit and update the source code, which will greatly reduce the conflict. At the same time, using Jenkins to automatically deploy the server-side package compiled code to the test and the online environment, eliminating the hassle of manually organizing the list of files to be published, but also avoid the release of files missing. In order to improve the development process quality, scientific and friendly normative development process, we choose Gitlab as the new Code warehouse, through branch management and code review to improve development efficiency, reduce publishing errors.

Two. Automating the deployment architecture Complete the function:

1. code Warehouse with Gitlab hosting, using Aoneflow Branch management mode (Ali named a branch of management mode, for reference to Gitflow, Githubflow and Gitlabflow).

2. after the source code is merged into the test branch, Jenkins automatically packages the compilation and deploys the compiled code to the test environment.

3. After the source code is merged into the publishing branch, Jenkins automatically packages the compilation and deploys the compiled code to the online environment.

4. make version tag for master Stable branch, and add tag version description.

5. Scaffolding and code separation, maintain a scaffold warehouse, provided to the various environments to compile.

Deployment Scenario Exploration

A. Jenkins merge code and compile, SSH send compiled code to test environment

Cons: Large amount of sending code, serious time consuming

B. Jenkins merge code and compile, compile results submit to GITLAB,SSH Connection test environment Update code from Gitlab

Cons: Post-compilation code merge into Gitlab conflict much, trouble

C. Jenkins Merge code, SSH Connection test environment Update Gitlab code, and then run the compile command

Advantages: Fast speed, less conflict

In summary, we choose Scenario C for the deployment code.

Post to line, cannot automatically trigger Jenkins build after merge to Release/prop release Branch, because I may have multiple feature branches at once to be published to the line, this time need to merge multiple feature branches into the publishing branch, You can then perform the build operation. So merging to the release branch and building the deployment to the line should be divided into two separate sections, respectively.

A picture wins thousands of words, combined with our actual development environment, the overall structure is as follows:

Three. Jenkins Configuration Combat

There are many options on the installation of Jenkins, which can be queried separately.

1. Install the plugin first:

• Gitlab Hook Plugin
• GitLab Plugin
• Publish over SSH

System Management--management plugin--optional plugin

2. New Job

3. Configure Git source

Click New Job, click Configuration-Source Management

Repository URL : Fill in the Git repository address

Click Add--jenkins

Choose SSH username with private key (need to generate SSH keys in the Jenkins server in advance)

• Username:root
• Enter directly: private key content or from a file on Jenkins master: private key storage path such as/home/role/.ssh/id_rsa
• Passphrase: password to generate SSH keys, if not set then do not fill

If you choose the private key content, you will need to gitlab your public key to Gitlab SSH keys:

Click Credentials to select the newly added certificate, if there is no red word error, prove that the setting is successful!

4. Set Webhook

Webhook as I understand it is an interface that can be triggered, which can be used to trigger a task under certain conditions.

The following options are found in the job configuration: If not, install the Gitlab Hook Plugin first

Copy the Webhook URL in the red box, open Gitlab If you fill in the Webhook address at the URL

Click Test after add, if prompted

The setup succeeds and Jenkins succeeds!

！！！ Attention:

Gitlab's Webhooks URL is set according to Jenkins Build permissions connection, if must log in to build must get Jenkins username and token, can be found in the Jenkins User-settings, URL format

http://<username>:<api-token>@<jenkins-server>/

If you do not need to log in to build it directly to the HTTP//jenkins-server/job/security_usm/build?delay=0sec , Security_ USM is the job name

So, if you are prompted with the following error:

Hook executed successfully but returned HTTP 403 <!doctype html>
You need to change the Webhook address in your Gitlab to the following form:
http://maoyu:[email protected]:8080/jenkins/project/front-end
5. SSH deployment to the server
Global configuration SSH Server: System Management-System settings
、
Once configured, click Test and the success appears to be successful.
Then configure the specific release content:

 
 

  
  
 
   
Path under Workspace in Source Files:jenkins
 
   
Remote directory: is a relative global configuration in publish over SSH--SSH server--Remote directory path
 
  
 
 
Source files can also be filled out, directly connected to the server after the EXEC command.
6. Display the time of the build log
If you want to display the following time in the console output
You can configure the following:
7. Merging Gitlab merge Request automatically
If you want to use Jenkins to automatically merge code after the merge request, you can skip this step if you don't need it, such as push Webhook directly to a branch.
In my current build scenario, both are available when publishing to a test environment, either by triggering an automatic merge from the merge request and then building it, or by manually merging to generate a push event trigger webhook. However, it is likely that after it has been submitted to the corresponding branch, it will automatically trigger Webhook,jenkins to merge into the corresponding branch and then compile the deployment. Look at the specific company's usage habits and some specifications.
Using plugins: Gitlab Merge Request Builder
This plug-in is determined by setting a branch and checking whether the branch has received a merge request request to determine if the build was made. Gitlab is an audit management, and when Jenkins builds, Gitlab merges the branches.
The configuration is as follows:
If you want to automate the merge operation after Gitlab pipeline is successful, you need to tick the following configuration, which can be said to be necessary, or you will not be able to complete the automatic merge:
The Gitlab corresponding merge interface has the following changes:
The initial interface is as follows, and you can close the merge request in the upper right corner:
In establishing a merge request to merge successfully, the following Red Cross hints may appear could not connect to the CI server. Don't worry about it, you can ignore it.
If the merge fails, the interface prompts as follows, and you need to go to Jenkins to view the log specifically why the merge failed:
Or you might see a hint like this:
The merge succeeds as follows, there will be some comment that Jenkins automatically adds in discussion:
Gitlab Merge Request builder triggers Gitlab pipeline without configuring CI/CD for Gitlab.
If in the Merge requests section of Gitlab, discussion does not have build started,build triggered, build finished Test Passed. These hints indicate Gitlab Merge The Request Builder plugin does not work, carefully check the configuration Gitlab Project path, and so on, if it is correct to try restarting Jenkins, re-launch the merge request.
If the Gitlab merge request builder under the configuration build trigger, click the Apply times below error, check the Gitlab Merge request Builder configuration in System administration-system Configuration Jenkins Username and Jenkins API tokens are correct.
Gitlab Merge conflict:
Click Resolve Conflict To edit the conflict manually, save to re-trigger the merge request, and then automatically merge
8. Parametric construction
Parametric construction is the first you need to manually click the Build button, second you can set some parameter variables to use in the construction, such as development environment, branch parameters and so on.
Why do I need parametric builds? Our current project is relatively simple, in fact, there is no need for parametric construction, but in the deployment to the master after the tag, encountered a bit of trouble, that can not give tag to set some personalized description of the copy, so look for the tag simply look at the version number may be difficult to know what to do in a version of the function. So when we posted to the line, we added a description of the feature to be published, which is used as a description of the tag deploymsg.
Parameterized builds can use the parameterized build process that comes with Jenkins, as follows:
Select string parameter in the join parameter, the first name is the variable name you want to add.
After the settings are set, the settings for parameterized builds are completed.
The second method can use Jenkins plug-ins, Jenkins has thousands of of various functional plug-ins, very rich, basic features we want to have plug-in support.
Here we use the plugin is: Generic Webhook Trigger
A ref variable is configured as configured above.

 
  
 Attached 1: How to know from Jenkins which branch of Gitlab Webhook is triggered?
See
A simpler approach is to gitlab the variables provided by plugin:
You can then print the following view in the shell:
echo "Gitlabbranch: ${gitlabbranch}";

 
  
 
Attachment 2: Generate multiple SSH keys for the same server:
You only need to name different names for the file names, as follows:

 
  
 
attached 3:jenkins timed syntax:
The task is performed every 2 minutes as follows:
Cron syntax has five bits, the meaning of the distinction is:

 
 

  
  
 
   
MINUTES MINUTES in one hour (0-59)
 
   
HOURS HOURS in one day (0-23)
 
   
Daymonth Day in a month (1-31)
 
   
Month month in a year (1-12)
 
   
DayWeek Day of the Week (0-7) where 0 and 7 is Sunday
 
  
 
 
Each of these fields can use some special characters in addition to the values in the range of values.

 
 

  
  
 
   
* Match all values in range
 
   
M-n matches all values in the M~n range
 
   
m-n/x or */x is built every X within the specified m~n range or throughout the active interval
 
   
A, b,..., Z matches multiple values
 
  
 
 
In order to generate timed tasks in the system, the symbol H (for "hash", followed by "hash" instead) should be used where possible, for example: Configuring 0 0 * * For more than 10 daily tasks will produce a large peak at midnight. In contrast, configuring H-H * * will still perform each task once a day, not all at the same time, to better use limited resources.
The symbol H can be used for the range, for example, H H (0-7) * * * represents 0:00 to 7:59 for a period of time. You can also use H to represent an interval with or without a range.
The symbol H can be considered a random value within a certain range, but it is actually a hash of the task name rather than a random function.
It is important to note that the day of the month-dom field, similar to */3 or H/3 short period due to the number of days of the month is not fixed, in most months will not work. For example, */3 will be within one months of the first day, the fourth day ... The 31st day executes, and the day of the next month continues to repeat. Hashing is generally chosen within 1-28 days, so H/3 will cause a gap within 3-6 days of running to the end of the month. (Long cycles will result in varying lengths, but this effect is also not obvious.) ）
Blank lines and lines beginning with # are considered to be comments.
In addition, @yearly, @annually, @monthly, @weekly, @daily, @midnight, and @hourly also support aliases. These use the hash system to automatically match, for example: @hourly and H * * * * * * * * * for any moment within one hours. @midnight actually represents a period of time between 0:00 and 2:59.
For example:
# every 15 minutes. (Perhaps: 07,: 22,: 37,: 52)
H/15 * * * *
# every 10 minutes in the first half hour. (3 times, perhaps: 04,: 14,: 24)
H (0-29)/10 * * * *
# every weekday from 9:45 A.M. to 3:45 P.M. at the end of this time interval of 45 minutes every 2 hours in the moment.
45 9-16/2 * * 1-5
#每个工作日从早上9点到下午5点这段时间内每间隔2小时之间的某刻. (perhaps at 10:38, 12:38, 2:38, 4:38)
H H (9-16)/2 * * 1-5
#每月 (except December) from number 1th to 15th at some point in the period.
H H 1,15 1-11 *

 
  
 Attached 4:jenkins shutdown and restart method: 1, close Jenkins
You only need to add exit after accessing the URL address of the Jenkins server. For example, my Jenkins address http://localhost:8080/, I just need to hit the http://localhost:8080/exit URL on the browser address bar to close the Jenkins service.
2. Restart Jenkies
Http://localhost:8080/restart
3. Reload configuration information
 Http://localhost:8080/reload

 
  
 
Attachment 5: Error Highlights: 1. Gitlab:the project were looking for could is found.
Fatal:could not read from remote repository.
Sure you have the correct access rights
and the repository exists.
Solution:
See if a git user has permission to submit to the Gitlab Warehouse
2.Could not open a connection to your authentication agent.
Need to manually open SSH, as follows;
Eval ' ssh-agent-s '
Execute Ssh-add again to
3. Exception when publishing, Exception message [Exec timed out or is interrupted after 120,001 ms
Adjust the time-outs appropriately, such as 300000
4. Ssh_exchange_identification:connection closed by remote host solution one. Change the SSH connection number to large
Modify this file on the server:/etc/ssh/sshd_config found this line # MaxSessions :
Remove the "#" from the front and change the number, then restart the sshd service sshd restart and reconnect.
Workaround two. Each time you exit SSH connection normally
Exit with the input "exit" after each execution of the command to prevent too many connections.

 
  
 Attached 6:SVN Batch detection submit shell script
See Github:https://github.com/saysmy/svnsh
Jenkins+gitlab Automated compilation deployment scenario exploration and service-side compilation Webpack combat