Standardization of Operation and Development

by YOUNGSTERXYF

Graduated at the end of March this year, Tencent to do operational development, has more than 6 months. At the time of entry, only 1 operational development colleagues were expanded to 5, plus 3 interns.

The operational development process at the time of entry is this: Write code on the Office Machine (Windows), after the function test passed, SSH remote connection to production server (Linux), vim open a new file, copy the code on the office machine, paste into vim, save, open the browser test online function /effect is correct, if not, edit the code file directly on the production server until the desired functional effect is achieved, and then copy the modified code from the production server to the Office machine (perhaps without this step, after which all the changes are made directly on the production server).

This process has the following problem: The code does not have version control without the same test environment code deployment process as the production server cumbersome Office development machine The code is probably older than the code on the production server. 4 points can cause chaos.

In addition, when the team expands from 1 people to more than one person, there is inevitably a problem of collaboration, and solving code development collaboration generally involves the following: Using code versioning to specify version-controlled Workflow coding specification project/Code document regular code review

In order to solve the above problems, I have done the following work: Build Gitlab server, test server

Personally, the 1th of the standardization of development work is versioning, which can accomplish a lot of automated tasks based on version control.

Usually personal code, documents are stored in the GitHub through Git version control, so choose Gitlab from the GitHub platform, server construction process see: Build test Server (source code compilation method).

The test server and the Gitlab server are the same machine, add the post-receive hook script in the hooks directory of a version library on the GIT server side, and when the code is submitted, the script is triggered to execute, and the script updates the code in each branch of the repository to a specific directory. The Nginx virtual host configuration is then added to root for each directory. So each time the code is submitted, you can open the browser to see the effect directly. Hook script code see: post-receive.py.

The implementation of the hook script is more violent---any code submissions will check out all the branch codes that are checked out.

So far, the Gitlab platform has been better used, but because the operating platform to be developed involves some monitoring data API problems, it is difficult to effectively use the test server, so the test server has not been formally used. Choose a git workflow

If a single git is used, then git's workflow is basically: git status,git add xxx, git commit-m "xxx", git Pull,git push Origin master.

When it comes to working with multiple people, the GIT workflow is more complex to reduce the amount of manual merge code and to avoid polluting git remote server version information as much as possible.

Our git workflow is as shown in the figure:

GIT's remote server-side and office-side version libraries maintain two shared branches: master and develop. The Master branch is a stable branch with the latest code consistent with the code on the production server. Develop is the test branch.

When a developer develops a new feature or fix bug, it first creates a unshared branch xxx from the Master branch in the local machine, writes code in the XXX branch, and requires the XXX branch to be merged into the develop branch when the test server tests the code. Push the develop branch to a remote server; After testing to determine that a production server needs to be online, merge the XXX branch into the master branch and push the remote server, where the production server deploys from the master branch pull code.

This workflow avoids the issue of version rollback caused by the "submit code First". Code moved to version control on the production server

Although the work is not very heavy, it is worth mentioning. Because can not affect the normal service, also can not cause any loss of file damage, so should be cautious.

I chose the weekend to complete the job at the company, as follows: Create a new version library on Gitlab, such as XXX, package the code on the production server back up; go to the Code root directory and execute git init;

Create a write. gitignore file, because some non-code files, such as pictures, videos, etc., do not need to enter version control, you should tell git to ignore. Note: Files that have entered version control, even if listed in. Gitignore, will not be ignored by git. Because there are more files, the directory structure is more complex, so it may be difficult to find all the files that need to be filtered, git add *, and then through GIT status to see which files have been added to the registers, if there is a need to filter the file is added to the registers, then perform git reset cancel Registers content, edit again. Gitignore;

Verify that the files that need to be filtered are matched by the contents of the. Gitignore to execute git add * && git commit-m "init";

Associate the local Git version library with the GIT version library on the Gitlab server side. Assume that the HTTP link for the Gitlab server-side version library is http://gitlab.server.com/xxx.git, executing git remote add Origin http://gitlab.server.com/xxx.git;

Push the local version library code to the remote Gitlab server side. Execute GIT push Origin master.

In this way, the code on the production server is moved into version control, and then the code is deployed, just pull (pull) from the GIT remote server.

The code deployment process was also intended to hook up---when the code was submitted to the Master branch, the hook script was triggered to pull the latest code from master branch onto the production server, but on the one hand, the test server could not connect to the production server because of network setup--- , this automatic approach is not used. After the code is committed to the Gitlab server side, you need to log on to the production server to perform Git pull. Project/Code Documentation

After a period of time, although several functional modules have been written for the operating platform, the implementation logic for the other modules associated with these modules is not clear. For modules that are not implemented by me, the problems that are specifically addressed, as well as the data sources of each module are not clear, there is no document to check, so set out to write a code document---explain the various functional modules to solve the problem, implementation logic, APIs, related cron scripts, data tables, description of the operating platform request processing process, User identity authentication method; The composition of the front-end page (because all the pages of the platform use an IFRAME combination structure), pointing out some hidden bugs in the code, "bad Taste" in the code, and proposing some programming suggestions for the problems of the Code;

For the front page HTML, CSS, JS, I do not have too much entanglements. Our front-end skills are not very well, the front-end implementation relies on bootstrap, jquery and other open source code base.

This work cost more time, resulting in a bit of a rush to work later, the document is also written more rough. Now, this document is very bad, only text narrative, no charts, more boring, of course, part of the reason is that the operating platform does not adopt a complete framework, each module through the front-end page piece together.

Some of the programming recommendations presented in this document have some merit:

1. Before the code is online, be sure to remove the debug statements, such as common.php:

File_put_contents ('/tmp/cron_test_ip.txt ', Print_r ($ip, true));

Debug statements are superfluous for code that is formally on line, can affect the performance of the application, and may also pose a security problem.

2. Clearly understand each line of code you write down.

For non-functional languages, every line of code may have "side effects," and if you don't understand the code you write, it can result in unpredictable consequences, and it will cost you a lot of debugging time, so don't use functions, classes, and language features that you don't understand very well.

3. A code "write three times".

Programming is the process of translating an idea in the brain into an executable program, and if the idea is not clear, it is impossible to write the correct program, even if the written program executes, there may be a logical loophole, a hidden bug. Writing a program is not coherent, but a repeated debugging process, the process is very concerned about the details, may cause you can not grasp the overall structure of the program, performance and so on, so after a program, the program can also be executed, you have to read and think again you write the program.

The first time: careful thinking to use the program to achieve the task logic, to avoid logic errors, loopholes;

Second: Write a program designed to implement an executable program;

Third time: Read the program completely again, remove unnecessary statements, optimize the structure of the Code, improve the performance of the program.

4. Properly handle cron script/api files.

Cron script/api files should be placed in the appropriate directory, and the Cron script/api files should be categorized using a more fragmented directory structure when the number of cron scripts/api files is large. In addition, discard code files should be purged in a timely manner.

5. The front page code as far as possible separate HTML, CSS, JS.

Be sure not to inline the CSS, that is, do not add CSS effects to the tags by using the HTML tag's style attribute, because it creates a lot of redundant code and affects the simplicity of the HTML code.

It is best to separate HTML, CSS, JS code into different files, that is, use the external reference JS, CSS. The advantages are 3:1) to improve HTML, JS, CSS code of their own concise readability, 2 can be in multiple pages (or even the entire site) between the sharing of JS, CSS, improve code reuse, reduce the workload of code updates, improve the consistency of the site style, 3 to facilitate the site content of static and dynamic separation, respectively, JS, The CSS code is compressed, and the CDN can be used to speed up the request response.

Step back, you can also in the HTML <style>, <script> tags to write css, JS code.

6. DRY (Don ' t Repeat yourself).

Do not copy the pasted code. When you find that the same code appears more than two times in different places, you should use functions, classes, public files and other means to avoid duplication of code, improve the reuse of code, but also for future updates to reduce the workload of the Code.

7. Avoid using operations that rely on a particular operating system or file system as much as possible.

For example:

(1) Avoid introducing files through absolute paths as much as possible, because code migration or change is much more intensive when there is a large number of absolute paths in your code, and you may miss changes, resulting in bugs. You can define a constant for the project root in a common file, introduce the file based on that constant if needed, and change the value of the variable only when the code is migrated or changed. In addition, if feasible, use relative paths whenever possible.

(2) Avoid directly invoking the shell commands provided by the operating system to accomplish the task as far as possible. On the one hand some special symbols for the shell is of special significance, if the parameters with these special symbols need special attention; On the other hand, for the portability of code, it is a big problem to call the shell command directly. In general, programming languages have libraries to support operations on objects such as file systems, and you should use them to accomplish tasks whenever possible. Standardize the propaganda and guidance

In order to implement the specification, need and the group operations development colleagues to explain the requirements of the specification, how to execute the specification (such as git use, git workflow), which things can be queried from the code documents, etc., so simple to do a share, PPT see: Standardized operation and Development. How to implement the "Regular Code review"

You know, I am a newcomer, it's hard to get colleagues to take the time to sit down and review the code at work, so I have to implement an automation tool---a cron script that executes once a week and gets a week's commit, For each of these committer randomly assign another committer all commits for the week as a review task, send a hyperlink on a commit in the task to the respective mailbox.

Code See: pack_git_commit.py

The TOFAPI imported in the code is my Python package for the Unified Messaging interface that the company provides. How to implement the "coding specification"

As with "How to implement the ' regular Code review '", I have also written an automated gadget to help implement the coding specification---a cron script that first customizes the git log command parameters, calls the git log command in the script, and parses the required data. Use different static analysis programs for different types of code files to analyze how each line of code does not conform to the coding style. The results are then submitted to a specific code base in Gitlab and a hyperlink to the result is sent to the mailbox of each person associated with the code file (who has made additions or deletions to the file) to provide an improved reference for the coding style.

The tool currently supports JS, PHP, python file Code analysis: JS Code analysis is based on Google JavaScript Style Guide as the standard, with the help of Google's closure_ Linter Tools for analysis (I hack the tool's code to support filtering the file represented by the regular expression file) the analysis of the PHP code is Zend Coding standards (http://framework.zend.com/wiki/ Display/zfdev2/coding+standards) as a standard, analysis of Python code with Phpcs is based on Pep 8, with Flake8 analysis

See main code: Codelintset.go.

Due to recent interest in Golang, the main program uses Golang implementations.

The send_mail.py involved in the code is also a mail-sending script using the Tofapi mentioned above, Template.tmpl is a golang template for generating static analysis results, including source code, analysis results, and related personnel three parts.

Attention:

There's still a problem with this program---git records two times per commit: Author date (file additions and deletions) and Committer date (commit time), but it is quite possible that a change after a commit to a local version library It took a long time to push to the remote server. The program is executed once a day on the Gitlab server, where the parameters of git log--since to Committer date. Even if a new commit is pushed to the Gitlab server within a day, the program executes the custom git log command, and the result is likely to be empty because it was pushed after a day of commit. After checking the API of Gitlab, it seems that there was no API for the time of the commit push, and it was estimated that you could analyze the Gitlab database and read the time of the commit push.

2013-10-22 Update

Codelintset.go has been updated to the second edition to resolve the above problem by reading the relevant data from the Gitlab database and putting the constants that were originally needed to be configured into the JSON file, which will parse the JSON file when the program runs. In this way, modifying the configuration does not require recompiling the source code.