Shell Development Specification

Version 1.0, refer to some articles on the web to be structured. Later intends to continue to revise. Complete a shell development specification that suits you.

Last modified: 2017.6.25

I. Naming specification 1, version and operating parameters

1) Note the version number before the script starts; (recommended)
2) If other tools are called, also specify the version number of the tool; (recommended)
3) Add the necessary running parameters for the script, similar to the operating parameters of the C program, you can use getopt to get the running parameter values, such as the basic parameters are:-V-version number;-h– help information; (recommended)

2. Variable naming

1) The name of the variable to be unified, it is recommended to use all uppercase letters, such as Apache_err_num, the semantics should be clear, can correctly express the meaning of variables, too long English words can be replaced by the first few characters. Multiple word connections using the "_" Number of connections, reference, must be referenced in ${apache_err_num} way;
2) Avoid meaningless characters or numbers: for example, the number 22 below does not know its exact meaning.

Count= 'grepfile'if  ' then do    Something  fi

3) Global variables and local variables:
I. If you need to use a global variable, prefix the variable with g, such as G_work_dir, when the variable is used, use {} to enclose it, that is, ${variable}
II. Use of local variables: scripts defined in a function (functions) variables we call local variables, must be declared in the local way, so that only in this function effective, so as not to cause in the function of the name and program in the names of variables, resulting in the normal variables in the program operation. Such as

function TestFunc () {    local I    for ((i=0; i<n;i++))    do         do something    done}    

4) Variable merging: When certain variables or configuration items are combined to make sense, such as the path and file name of the file, it is recommended that the variables to be combined be combined to assign a new variable, which facilitates subsequent calls and facilitates subsequent modifications.

3. Function naming

1) The function name is capitalized with the first letter of the word, such as CreateFile, and is semantically clear, such as Dumpdatatofile () instead of datafile (). At the same time to pay attention to the prefix, such as the suffix max is the maximum, Min is the minimum value, the prefix is a judgment function, get is a value function, which is beneficial to the return of the function clearer understanding.
2) Note that in the case of abbreviations, the initials are also capitalized, such as gethtmlcontent instead of gethtmlcontent, which facilitates the unification of naming conventions.

4. Script (module) naming

1) The shell script uses the unified suffix:. sh
2) the start and stop scripts for all modules must be uniformly named start_{module name}.sh and stop_{module name}.sh
3) The naming of modules (and their scripts and binaries) should represent their features and functions, and they are forbidden to be named in the form of personal initials.
4) scripts and binaries in the module are named forbidden and duplicate names of other scripts and binaries.
5. Naming of temporary files
Try to avoid temporary files, if you must use a temporary file, use the PID of the script as a suffix, and clear the temporary file at the end of the script. For example:

Pid=$ $TMP _file="Tmp_keywords." ${pid}grep keywords files > ${tmp_file} ... #endrm ${tmp_file}

Two, Code style 1, code framework

1) The interpreter declaration is best for #!/bin/sh (recommended), and you should pay extra attention to the execution environment if you are not joining the crontab scheduled task.
References to profile/library functions source foo_conf.sh
2) The main process only implements the program backbone, the function implementation should be encapsulated in the sub-function;
3) for scripts that can execute independently, use the usage and version functions to output script usage and versioning information (recommended)

2. Functions

1) Add function reserved words to function names when defining functions
2) The Display function returns: Each branch of the function contains the return statement that is displayed in each of the branches of the function, and the returned value is followed. Even a function that does not care about the return value may inadvertently judge its return value and perform a series of actions on subsequent calls, which we should be aware of at the outset that the displayed write return statement does not bring much burden, but instead, it makes the logic of the function clearer and more rigorous.

3. Conditional statements and Loops

1) When using conditional statements and loops, try to use a uniform format instead of using ";" Separated

if [-D ABC]  Then Do     somethingfiwhile1  ]  Do Do     somethingdoing for in *do     somethingdone 

2) Try to use one statement per line instead of ";" Separates multiple statements. Make as many judgments as possible about the success of the operation and handle it accordingly, such as Dosth && Doright | | Dowrong. This simplifies the use of statements and makes semantics clearer;
3) To use a simple statement. In particular, multiple-pipe commands should be avoided, and multiple-pipe commands need to focus on each management message each time it is read, until the last command, if it is split into multiple statements, is highly readable;
4) to undertake the above 3rd point, the script often has grep cut awk sed and other commands together with complex statements. You should consider simplifying statements as much as possible. For example, multiple grep awk operations can often be replaced by 1 sed.

4. Indent

When you use conditional statements, you use an indentation, which is the TAB key, for Each loop or inner operation.

if [-D ABC]  Then CD ABC     if [-D BCD]      Then         dosth    fiCD. fi

III. Note Specification 1, document/module description

Explain the main purpose of the module, version information, input and output files, dependent tools and their version information, before and after the process script (optional), the format is unified, do not make mandatory requirements, can be as follows:

##!@TODO: URL Analyse # #! @VERSION:1.0 ##!@AUTHOR: MM; BB # #! @FILEIN: data/url.crawl##!generated by dedup_crawl.sh, in the form of ... # #! @FILEOUT: result/good_grp##! No problem by detecting the alias group ##! @FILEOUT: result/wrong_grp##!failed to pass detection of problematic alias Group # #! @DEP:wget 1.10.2 ##! @DEP: Lftp3.0.6 ##! @PREV: Dedup_crawl.SH ##! @NEXT: Dedup_update.SH

The path given in the note must have a relative path relative to the root directory that was generated when the module was installed.
Example
The module Seek_url is installed in the path directory, the installation becomes the new directory seek_url, then all relative paths are relative to path/seek_url/.

It is recommended to write comments for the script's configuration and scripts before and after the script in the process. To facilitate testing and new familiarity with the module process. Recommended

example:##! @conf Conf/config_dedup_check. SH ##! dedup-Check Module-wide configuration file # #! here the main read BinPath and Alias_file set the script name in the process flow before this script # #! @prev Dedup_crawl. SH The script name in the processing flow after this script # #! @next dedup_update. SH

All scripts must be written in a script or configuration file: dependent modules, tools, scripts (required)

example:##! @dep spuser-ftool##! Brief Description # #! @dep read_di##! Brief description

2. Important Function Description

For important functions, it is necessary to explain the function use, parameters, return values, author, version, format does not make mandatory requirements, can be as follows:

hostname  ##! @AUTHOR: Somebody # # 1.0  ##! @IN: $1 + = IP # #! @IN: $2 = Port # #01 = failure

3. Other

The script must have very detailed comments, including the definition of variables, function definition, return value of the letter, the purpose of each step, and so on, especially to strengthen. Note Use standard English or detailed Chinese instructions as much as possible.

Four, the log specification

1. The script and the C program, need to have the logging script's running state, the operation process and so on. Recommendation: Each log begins with the current time of the record, then the log description information of the record, including the level of the log, such as Fatal,warning,notice, and the full, unambiguous contextual information, especially the fatal and WARNING logs.
2. Each script must have its own log file, do not log multiple scripts to the same log file, even if there is a dependency between the scripts can not.

V. Specification of interface files

1. The interface file between all modules must generate the MD5 checksum code at the same time as the generated one, and a file corresponding to a MD5 file, the MD5 file is named after the interface file name plus ". MD5".
2. The data copy between machines is recommended to use wget, minimizing the use of SCP and SSH. But as long as the wget gets remote data, you must use MD5 to verify the integrity of the data.
3. All interface files are recommended for format checking and should be consistent with empty files and non-existent file judgments.
4. Recommended interface files It is advisable to have a corresponding flag file indicating the generation time for inspection and fault tolerance.

Vi.. Configuration specification

When multiple scripts work together, you should try to use common function libraries to accomplish approximate functions, use configuration files to unify variables and references of external data, so as to facilitate future function upgrade and environment change operation. If you are using a common library of functions, we recommend that you add the path to the common library script in the configuration file, even if you are configuring it in the current directory, such as filename_path= "./", the common library of functions in the script and the configuration file using the source ${filename_path}/ filename.sh mode load.

1. Configuration files

1) All scripts must clearly separate the function and configuration;
2) configuration file to name the conf_ script name. SH (recommended);
3) In principle, the configuration item is placed under the directory conf/and is loaded by the source conf/conf_spdata.sh;
4) The naming specification of the configuration item is the same as the variable naming specification, which requires that all configuration items should have a value, avoid the presence of empty content items, and the contents of the configuration items are enclosed in double quotation marks, such as:
Output_file_path= "./output"
5) Configuration of path parameters: The path of the configuration path, local path, or remote machine is often required in the script. The configuration of the path is strongly recommended to use the global path, the configuration of the global path is recommended to start with "/", there is no "/" at the end, when using the path configuration item in the script, the different configuration items, variables or wildcards must be separated by "/" and No "/" can be omitted.
6) The configuration of variable parameters: variable parameters mainly refers to the status report and the recipient of the alarm, must be written in the configuration.
7) machine name, port and user name reference: All online scripts must be configured to reference information such as machine name, port, and user name.
8) Binary reference: When a module needs to invoke a binary program/script that is not maintained by this module, it must be referenced indirectly through a configurable path, prohibiting the direct copying of the binaries of other modules to its own directory.

2, the configuration item check

1) in the script to check whether the configuration item is empty, especially some important, affect the following script normal operation of the configuration items, you must make an empty check, avoid errors such as missing in the configuration file, check the format can be as follows:

if [-Z ${output_file_path}]  Then Write log fi      

Seven, Alarm specification

1) The title of the alarm message (and SMS alarm) must be in the following format:
[Alarm level] [Service Name] [Machine name] [Module name] Alarm information [TIME]
Example:

Echo " [error][image][$ (hostname-s)][${0}][urlmerge for group ${group_no} failed][$ (date + '%y-%m-%d%h:%m:%s ')] "

2) The sender of the alarm message must be real, the alarm message must include the sender machine name. Do not allow to issue on the a machine on the name of the B machine on behalf of the alarm, recommended the use of ' hostname ' command

3) The recipient of the Online module alarm message (or SMS) must include the module owner, Supervisor, OP, and disable the module alarm (or SMS) to the whole group. Each alarm containing the same information shall not be more than 3 in principle. Must

Viii. Other Details

1) involving dependent machines that need to be configured to appear in the script's configuration file, prohibit the definition of machines anywhere in the script, which is not easy to maintain and error-prone;
2) must have basic log output; (required)
3) To pay attention to the efficiency of the script and system consumption, comprehensive, balanced consideration; (required)
4) function parameter passing: When a function is called, the arguments passed to the function are passed as a variable, and the variable must be enclosed in double quotation marks, in order to prevent a variable from having more than one space-delimited field, causing the function to be mistaken for multiple parameters, such as the configuration entry for alarm messages and alarm text messages, using the following method:

Alert "Some Thing" "${log}" "${alert_mail}" "${alert_mobile}"

5) avoid large cat files. For example, for the ID in ' Cat id_file ';d o ...; Done Instead, the file is read in readline form. Must
6) All previous orders require the IF [$?] to determine the return value, and for the exception branch, there must be a corresponding processing policy: either print the warning log, or the alarm exits/retries. Must
7) When the sort exceeds 1G data file, you must specify its temporary folder with-T, it is recommended to use-T to specify the TMP directory to itself when using the sort. Must
8) Avoid using large while/for loops, and if necessary, consider using the awk command instead. Recommended
9) need to configure Crontab script, note that the crontab time may have 1 minutes before and after the fluctuation, for the time accuracy of the more stringent requirements of the script, not recommended for example 00 minutes such a moment.
10) for a series of commands with strict dependencies, it is recommended to use && for example: Make Mydir && mv myfile Mydir, and disable background Run & command for scripts with pre-order. Recommended
11) before and after the script is run, be careful to purge out-of-date data (last run generated data) (recommended)
12) Before the script starts running, you need to specify the current running path, such as cd/home/img/img, and you must also identify the data generation path, such as data/spdata_stat/splendid.list; (required)
13) For more complex scripts, consider using some functions to encapsulate the function points. This will make the script legible and readable; (recommended)
14) Any error must be printed to the log, and serious errors must be sent in the form of a mail or SMS alert. Must
15) for logic-complicated scripts, you can use Set-x to print command execution for easy debugging and troubleshooting (recommended)
Before wget, be careful to delete local files first; (recommended)
17) When copying larger files, it is a good idea to put CP into a temporary folder and then MV to the target folder. This prevents the downstream module from reading the incomplete files. Recommended
MV,CP and other commands, note the use of recursive and mandatory commands such as –r–f (required)
19) In the script, be careful to escape the single quotation mark ' double quotation mark '. If you do not explicitly escape the meaning, it is best to be careful with the unit self-test. Recommended
20) When using the Paste command, be aware that the file is delimited with tab or space.
21) When using PS Axuw|grep to obtain information, note the maximum width of PS printing. Recommended more dozen W, such as PS Axuwwww|grep ... ; (recommended)
22) When using commands such as Sort,uniq,join,comm, be aware of two points: 1, whether the order is required (for example, Uniq,join,comm requires the input file is ordered), 2, if ordered, what sort of order is required (for example, Comm requires that the input file is a dictionary order, Rather than numerical order); (recommended)
23) When the field matches the log, be aware of special characters, such as

spurl[stargirls]spid[2534969]opusr [stargirls]opuid[6545440]

Such a log, if only using [and] as delimiters, there is a problem, because Spurl may contain [or] and other special characters, resulting in a subsequent match all error.
24) Special signal Processing: When the script is running, we sometimes temporarily interrupt the script (Ctrl + C), if the script is doing multiple file operations, and generate some temporary files, then we need to break the script, we have to recover or clear some process files. In complex scripts, special processing is needed for some of the aborted signals.

Reference:

This article address: http://www.cnblogs.com/chinas/p/7073061.html, reproduced Please indicate the source, thank you!!!

Shell Development Specification