How to test the correctness of shell scripts

Blog has been migrated, new address

Bytes ----------------------------------------------------------------------------------------

In practice, you need to test the correctness of shell scripts.

How can we test in the fastest and most effective way?

A lot of development habits are as follows: after writing/getting it, you can run it to check whether the input, output, and desired operations are completed.

In fact, this is not rigorous. If QA is not performed, the risk is still quite high.

The following is the shell script test process, which is for reference only.

1. code reading:

After writing or getting a shell script, you don't have to rush to run it. Although practice is the only criterion for checking the Code, many low-level bugs can be avoided during the code reading process.

Read what?

A. Code logic. What is this script used for? What are the main steps and what are done separately?

Used to check whether any missing logic exists or is not required.

B. Specific syntax, variables, and judgment statements

In terms of syntax, whether the variables are defined, whether the statement logic is correct, whether to consider exceptions, whether errors exit, and whether to return correct status values.

2. syntax check:

Shell syntax is quite speechless, and many things are easily overlooked.

Command Format: Sh-N ***. Sh

If no exception is output, the script has no obvious Syntax problems.

3. Run tracking:

Practice is the only criterion for testing and sorting.

However, instead of directly running and looking at the final result, many intermediate processes will be omitted.

Command Format: Sh-VX ***. Sh

The result is as follows:

We can see that

Original command of each line of code (none +): [This is the effect of-v]

Code execution conditions (with +), including calculation results, logical judgment results, variable assignments, and so on [-x results]

What we need to pay attention to is this information, mainly the variable value and the logical judgment result.

4. Overwrite Branch:

Directly run the task and only cover the main process. The Code of other control flow branches cannot be overwritten.

For the key logic, we need to create conditions so that the running script can enter the corresponding branch.

5. Others:

A. About bashdb:

You can try it, but the input-output ratio is not high.

B. unit tests:

In actual work, due to the high project pressure, the unit test cost is still quite high, so far there is no.

6. Is there a better way?

Okay, single-step tracking. The script is short, but there won't be too much log information. If there is more, there will be calls to other scripts and so on .....

It is easy to handle thousands of logs.

The tracked shoes share the same feeling. They are not friendly enough. They are pale and pale, with one thousand lines coming down and being dazzled.

It is easy to omit (LZ has been pitted for many times. If you read or do not read the... error message, it is clearly there, that is, turning a blind eye)

So. A layer of optimization is performed to process logs and use regular expressions to mark information I care about.

Comparison:

After processing: (mark the error and key information in color, which can be displayed on a Linux terminal)

The script is implemented in Python, Location: https://github.com/wklken/pytools/tree/master/shell

The idea is: Execute, capture all logs, use regular expressions for matching, color them, and output

We welcome optimization to make it more functional.

Code:

#! /Bin/ENV Python #-*-coding: UTF-8-*-# @ Author: wklken # @ mail: wklken@yeah.net, lingyue.wkl@taobao.com # @ Date: 20120706 # @ version: 0.1 sh-N, check for static syntax #0.2 sh-VX, color the output log which I care about #0.2.1 rebuild all functions, lines 200 +-> 120 #0.2.2 refine the re pattern. #0.2.3 add sh Params support. fix bug and add re patterns #0.2.5 add warn/error pattern and collect the resu Lt #0.2.6 use decorator to refine print, refine collect method # @ Desc: Quick test shell script. the target is hacking into it and get all the status I need. # todo: Need to keep source code in 200 lines! Refine! Import sys, osimport commandsimport re # color definedcolor_none = "c_none" color_green = "C_G" color_red = "c_r" color_yellow = "c_y" color_purple = "C_p" color_blue = "C_ B" color_map = {color_none: "\ 033 [M", color_green: "\ 033 [01; 32 m", color_red: "\ 033 [01; 31 m", color_yellow: "\ 033 [01; 33 m ", color_purple:" \ 033 [01; 35 m ", color_blue:" \ 033 [01; 34 m ", none: "\ 033 [M"} # The command used definedsh_n = "Sh-n" Sh_x = "Sh-VX" log_begin = "Export PS4 = '+ $ {bash_source} | $ {lineno} | $ {funcname [0]}-> '; "log_begin =" "# The type of output log lineline_type_cmd =" cmd "line_type_exc =" exc "line_type_cmt =" CMT "cmd_y = color_map.get (color_yellow) +" cmd: "+ color_map.get (color_none) # ---------- pattern used to match begin ----------------------------- #0. specialpattern_addsign = Re. compile ("(^ \ ++)") #1. execute comma Nd log match patternexc_mark_pattern = [(R "([\ [\])", color_yellow ), # For condition testing must be the first one (R "([12] \ D {3}) (1 [12] | 0 [1-9]) (0 [1-9] | 1 \ d | 2 \ d | 3 [01]) ", color_purple), # date yyyymmdd (R" (tbsc-Dev )", color_red), # path: tbsc-Dev (R "([A-Za-Z _] [a-zA-Z0-9 _] * = [\ s | \" \ "] *) $ ", color_red), # Params = none (R" (exit \ s + -? \ D * | return \ s + -? \ D *) ", color_blue ), # exit status (R "(\ s (\-[acbdefgnorsuwxzl] | \-(LT | le | GT | Ge | EQ | ne) \ s)", color_yellow ), (R "(\ s (=|=|<=|>=|\+ =|<|>| '! = '| \ &) \ S) | '! ') ", Color_yellow), (R" (\ s (\-input | \-output | \-I | \-O) \ s) ", color_yellow),] exc_mark_pattern = [(Re. compile (s), color) for S, color in exc_mark_pattern] #2. error/warn result log match pattern # 100% errorerror_mark_pattern = [(R "(no such file or directory | command not found | unknown option | invalid option)", color_red ), # result-> file not found (R "(unary operator expected)", color_red), # test failed (R "(PE Rmission denied) ", color_red), (R" (syntax error | unexpected | read error) ", color_red), (R" (Java. io. filenotfoundexception | Org. apache. hadoop. mapred. invalidinputexception | Java. lang. illegalmonitorstateexception) ", color_red), # javaerror] error_mark_pattern = [(Re. compile (s), color) for S, color in error_mark_pattern] # may be not error, just warn, noticewarn_mark_pattern = [] warn_mark_pattern = [(Re. compile (s), Color) for S, color in warn_mark_pattern] #3. command log match pattern1__mark_pattern = error_mark_pattern + warn_mark_pattern + \ [(R "(line \ D +)", color_red ), # error report the line No (R "(\ $ (\ {\ W + \})", color_purple), (R "(\. \.) ", color_purple), # Relative Path (R "((? <! -) \ B (\ W +) \ B =) ", color_yellow), (R" (\ $ (\ W +) ", color_purple ), # variable name (R "(\ W + \. sh \ s *) ", color_green ),#*. sh (R "(')", color_green), # ''(R" (\ s? \ W + \ s * \ (\) ", color_green), # function () (R" (\ {\ s * $ | ^ \} \ s * $) ", color_green), # function {} (R" (^ export \ s | ^ source \ s) ", color_yellow), (R" (\ |) ", color_green ), (R "(<| >>|< |>)", color_yellow),] pai_mark_pattern = [(Re. compile (s), color) for S, color in character _mark_pattern] # ---------- pattern used to match end character # static Params definederror_lines = [] # functions begindef str_coloring (str_info, color = color_none): "color Str" Return color_map.get (color, color_map.get (none) + str_info + color_map.get (color_none) def print_symbol (str_info ): "print the symbol" print "-" * 20 + str_info + "-" * 20def wrap_print_func (ARG): "Wrap func, print begin and end sign "def newfunc (func): def newfunc_withparams (* ARGs, ** kwargs): print_symbol (Arg +" begin ") func (* ARGs, ** kwargs) print_symbol (Arg + "end") return newfunc_withparams return newfunc @ wrap_print_func ("static Syntax") def static_syntax_check (file_path ): "Check the static Syntax" cmd = sh_n + file_path result = commands. getoutput (CMD) If result: Print "script syntax check:" + str_coloring ("failed", color_red) print str_coloring (result, color_red) else: Print "script syntax check: "+ str_coloring (" pass ", color_green) def pre_handler (result):" pre handle the result lines "" Pass @ wrap_print_func ("process Log check ") def dynamic_log_process (file_path, Params): "process the log of SH script" cmd = log_begin + sh_x + file_path + "" + Params result = commands. getoutput (CMD) pre_handler (result) process_line (result) def define _type (line): "" return the type of line, and can do something with it "If line. startswith ("+"): Return line_type_exc, line Elif line. lstrip (). startswith ("#"): Return line_type_cmt, line else: # Return line_type_cmd, 1__y + line return line_type_cmd, linedef mark_sign_by_pattern (line, line_type = line_type_exc ): "mark the STR by pattern" # Can't use in py2.4, ni Mei a # use_pattern = exc_mark_pattern if line_type = line_type_exc else pai_mark_pattern if line_type = line_type_exc: use_pattern = exc_mark_pattern else: use_pattern = pai_mark_pattern native_line = line for PT, color in use_pattern: M = pt. findall (line) If M: line = pt. sub (color_map.get (color) + R "\ 1" + color_map.get (color_none), line) for PT, color in error_mark_pattern: E = pt. findall (native_line) if E: error_lines.append (line) return linedef process_line (result): "" format each line. with the pattern "lines = result. split ("\ n") for line in lines: line_type, line = pai_type (line) If line_type = line_type_exc: Result = mark_sign_by_pattern (line, line_type) print inline (color_map.get (color_green) + R "\ 1" + color_map.get (color_none), result) Elif line_type = line_type_cmd: Print inline (line, line_type) Elif line_type = line_type_cmt: print Line @ wrap_print_func ("result collect") def warn_error_collect (collect_list, collect_type = "error"): Print str_coloring ("result type:" + collect_type, color_green) if (collect_list): Print str_coloring (collect_type + "found:", color_red) + str_coloring (STR (LEN (collect_list), color_yellow) for line in collect_list: print line else: print str_coloring ("no" + collect_type + "found", color_green) ARGs = sys. argv [1:] sh_name = ARGs [0] Params = "". join (ARGs [1:]) static_syntax_check (sh_name) dynamic_log_process (sh_name, Params) warn_error_collect (error_lines, "error ")

Okay, that's all.

The tool is implemented to improve efficiency and save time.

The end!

Wklken

Gighub: https://github.com/wklken

Blog: http://wklken.me/

2012-09-15

Please indicate the source for reprinting. Thank you!