Step-by-step process examples of improving program quality

Step-by-step process examples of improving program quality

 

How can we improve the program quality? This section describes common quality attributes and implementation strategies of programs in the Web server software service quality overview, this article describes how to gradually improve program quality through a batch rename tool for image files implemented by Python.

The batch rename tool for image files is used to rename images under the specified directory/home/user/path/to/photos/(xxx.png,yyy.png) to prefix0001.png, prefix0002.png ,...

 

Prototype

First, you can compile a basic available program batchrename_basic.py. This program is not perfect, but can complete the initial task. Note that a closure is used to generate a number, which is used to separate the process of generating the number from a reusable process. However, this process cannot predict the list to be generated, therefore, only one serial number is returned each time. The program is as follows:

# -*- coding: cp936 -*- import os import os.path as PathUtildef createDesignator(num, bits):    return str(num).zfill(bits)def number_generator(start_num=0, bits=4):    start = []    start.append(start_num)    def inner():        start[0] = start[0] + 1        return createDesignator(start[0], bits)    return innerdef batchrename(dir_path, prefix="IMG_",generator_func=number_generator()):    '''    rename files (such as xxx.[jpg, png, etc]) in the directory specified by dir_path to [prefix][designator].[jpg, png, etc], designator is generated by generator_fuc    '''    names = os.listdir(dir_path)    for filename in names:        old_filename = PathUtil.join(dir_path,filename)        if PathUtil.isfile(old_filename)==True:             newname=prefix.upper() + generator_func() + '.' + getFileSuffix(filename)            os.rename(old_filename,PathUtil.join(dir_path,newname))def getFileSuffix(filename):    try:        sep_ind = filename.index('.')        return filename[sep_ind+1:]    except ValueError:        return Nonedef testGetFileSuffix():    assert getFileSuffix("good.jpg") == "jpg"    assert getFileSuffix("good") is None    print "testGetFileSuffix Passed."def testNumberGenerator():    geneNums = []    generator = number_generator()    for i in range(10):        geneNums.append(generator())     assert geneNums[0] == '0001'    assert geneNums[1] == '0002'    assert geneNums[9] == '0010'    print 'testNumberGenerator Passed.'if __name__ == '__main__':    testGetFileSuffix()    testNumberGenerator()        dir_path = '/home/lovesqcc/setupdir/scitools/pic/mmnet/beauty'    batchrename(dir_path, prefix="beauty_")

 

　　Robustness

Robustness reflects the program's ability to respond to errors. An APP that requires network connection runs smoothly under normal network conditions. What if there is no network? The user must be notified to connect to the network first. Or use the input to automatically correct the error. For example, if you search for jquery in a search engine, you accidentally write it as jqeury. The search engine will prompt you if you want to search for jquery. In this example, an error is reported when the path does not exist.

Traceback (most recent call last):  File "batchrename_robust.py", line 57, in <module>    batchrename(dir_path, prefix="beauty_")  File "batchrename_robust.py", line 21, in batchrename    names = os.listdir(dir_path)OSError: [Errno 2] No such file or directory: '/home/lovesqcc/setupdir/scitools/pic/mmnet/beauty'

The solution is simple: extract names = OS. listdir (dir_path), write it as a function, capture exceptions, and rewrite the row to names = getDirFiles (dir_path ):

def getDirFiles(dir_path):    try:        return os.listdir(dir_path)    except OSError, err:        print 'No Such Directory: %s, exit.' % dir_path        os._exit(1)

 

Customization

If you want to specify the path and prefix, you must modify and redeploy the program, which is obviously stiff. The console program usually needs to add the command line parameter, while the actual application uses the configuration file. The following describes how to use the argparse module to add command line parameters to the program to make it customizable. Add a parseArgs method and modify the main method. Note that tuples are used to clearly express the format of parameters to be returned, which is convenient for the main program. Magic numbers are expressed by string constants to ensure maintainability.

Usage: $ python batchrename_robust_customized.py/home/lovesqcc/setupdir/scitools/pic/fuzhuang/fz2/-p fz2-m NUM 1 5

-P and-m are optional. By default, you only need to specify the directory path.

import argparseDEFAULT_PREFIX = 'IMG_'DEFAULT_START_NUM = 1DEFAULT_BITS = 4NUM_METHOD = 'NUM'def parseArgs():    description = 'This program is used to batch rename files in the given DIRECTORY to PREFIX_GeneratedDesignator. GeneratedDesignator is a BITS number counting from START_NUM to the number of files (etc. PREFIX0001,PREFIX0002,...) in the given DIRECTORY with leading zero if necessary.'    parser = argparse.ArgumentParser(description=description)    parser.add_argument('DIRECTORY', help='Given directory name is required')    parser.add_argument('-p','--prefix',nargs='?', default="IMG_", help='Given renamed prefix')    parser.add_argument('-m','--method',nargs='*',help='method to generate designator, etc --m [NUM [START_NUM [BITS]]]')    args = parser.parse_args()    dir_path = args.DIRECTORY    if args.prefix:        prefix = args.prefix    else:        prefix = DEFAULT_PREFIX    if not args.method:       method = NUM_METHOD       start_num = DEFAULT_START_NUM       bits = DEFAULT_BITS       return (dir_path, prefix, (method, start_num, bits))    if type(args.method) == list:        if len(args.method) == 0:            method = NUM_METHOD            start_num = DEFAULT_START_NUM            bits = DEFAULT_BITS        elif args.method[0] ==NUM_METHOD:            method = NUM_METHOD            if len(args.method) == 1:                start_num = DEFAULT_START_NUM                bits = DEFAULT_BITS            elif len(args.method) == 2:                start_num = int(args.method[1])                bits = DEFAULT_BITS            elif len(args.method) == 3:                start_num = int(args.method[1])                bits = int(args.method[2])        return (dir_path, prefix, (method, start_num-1, bits))

if __name__ == '__main__':    testGetFileSuffix()    testNumberGenerator()        (dir_path, prefix, (method, start_num, bits)) = parseArgs()    if method == NUM_METHOD:        number_generator = number_generator(start_num, bits)    batchrename(dir_path, prefix, number_generator)

 

　　Traceability

Traceability reflects the predictability and monitoring of the program running process. Record the key status and path of the program running, which is also very helpful for debugging when an error occurs. In this example, to record the specific information of the file rename, simply print the following information in the program:

os.rename(old_filename,PathUtil.join(dir_path,newname))print '%s rename to %s.' % (filename, newname)  # should be info log

Security

Security usually expresses two meanings: 1. The program cannot damage user data; 2. the program must prevent other programs from damaging user data or snoop user privacy. The first one cannot be violated. When we repeatedly run $ python batchrename_robust_customized.py/home/lovesqcc/setupdir/scitools/pic/fuzhuang/fz2/, we are surprised to find that the file becomes fewer after renaming! When enough times are run, only one file may be left! What's going on? After running several times, the following result is taken:

IMG_0006.png rename to IMG_0002.png.IMG_0003.jpg rename to IMG_0003.jpg.IMG_0002.png rename to IMG_0004.png.IMG_0005.jpg rename to IMG_0005.jpg.IMG_0007.png rename to IMG_0006.png.

After analysis, you can see that Python OS. rename overwrites existing files on UnixSystem by default, and OS. listdir outputs unordered results! The solution is also very simple: first sort the OS. listdir output results and then rename, that is, to modify getDirFiles:

def getDirFiles(dir_path):    try:        filenames = os.listdir(dir_path)        filenames.sort()        return filenames    except OSError:        print 'No Such Directory: %s, exit.' % dir_path        os._exit(1)

 

Reusability

The key to reusability is the single responsibility principle and interface definition orthogonal. A single responsibility principle refers to the fact that a function or method is only a small task, with good name and meaning. The interface definition orthogonal means that everything defined by each function or class interface does not overlap, it can be combined to implement very flexible functions. If the program has good reusability, it will also benefit from the extension and affect localization. When writing a program, you should always consider extracting reusable processes and methods. Reusability also helps to write more effective unit tests.

Scalability

Scalability reflects the ability of the program to respond to changes in requirements. In this example, the scalability is embodied in two points: 1. recursively rename the subdirectory of the Directory; 2. batch rename multiple directories with different prefixes; 3. different numbering methods are supported. For the first point, you only need to modify the batchrename method. If the directory is detected, batchrename is recursively called. For the second point, you need to modify the command line parameter format and add the-d parameter, the number of parameters must be at least one. Modify the-p parameter to zero or more parameters. If the number of given directories is greater than the given prefix, the last prefix is used to supplement the number of prefix. If the number of given directories is smaller than the number of prefix, the extra prefix from the last prefix is ignored. Modify parseArgs and main. The final procedure is as follows:

$ Python development-d/home/lovesqcc/setupdir/scitools/pic/fuzhuang/fz2 // home/lovesqcc/setupdir/scitools/pic/fuzhuang/fz1-p fz2 fz1-m NUM 1 5

# -*- coding: cp936 -*- import os import os.path as PathUtilimport argparseDEFAULT_PREFIX = 'IMG_'DEFAULT_START_NUM = 1DEFAULT_BITS = 4NUM_METHOD = 'NUM'def parseArgs():    description = 'This program is used to batch rename files in the given DIRECTORY to PREFIX_GeneratedDesignator. GeneratedDesignator is a BITS number counting from START_NUM to the number of files (etc. PREFIX0001,PREFIX0002,...) in the given DIRECTORY with leading zero if necessary.'    parser = argparse.ArgumentParser(description=description)    parser.add_argument('-d','--directories', nargs='+', help='Given directory name is at least one required')    parser.add_argument('-p','--prefix',nargs='*', help='Given renamed prefix')    parser.add_argument('-m','--method',nargs='*',help='method to generate designator, etc --m [NUM [START_NUM [BITS]]]')    args = parser.parse_args()    dir_path_list = args.directories    dir_num = len(args.directories)    if not args.prefix or len(args.prefix) == 0:        prefix_list = [DEFAULT_PREFIX] * dir_num        prefix_num = dir_num    else:        prefix_list = args.prefix        prefix_num = len(args.prefix)    if prefix_num > dir_num:        prefix_list = prefix_list[0:dir_num]    else:        prefix_list.extend([prefix_list[prefix_num-1]]*(dir_num-prefix_num))    if not args.method:       method = NUM_METHOD       start_num = DEFAULT_START_NUM       bits = DEFAULT_BITS       return (dir_path_list, prefix_list, (method, start_num, bits))    if type(args.method) == list:        if len(args.method) == 0:            method = NUM_METHOD            start_num = DEFAULT_START_NUM            bits = DEFAULT_BITS        elif args.method[0] ==NUM_METHOD:            method = NUM_METHOD            if len(args.method) == 1:                start_num = DEFAULT_START_NUM                bits = DEFAULT_BITS            elif len(args.method) == 2:                start_num = int(args.method[1])                bits = DEFAULT_BITS            elif len(args.method) == 3:                start_num = int(args.method[1])                bits = int(args.method[2])        return (dir_path_list, prefix_list, (method, start_num-1, bits))def createDesignator(num, bits):    return str(num).zfill(bits)def number_generator(start_num=0, bits=4):    start = []    start.append(start_num)    def inner():        start[0] = start[0] + 1        return createDesignator(start[0], bits)    return innerdef getDirFiles(dir_path):    try:        filenames = os.listdir(dir_path)        filenames.sort()        return filenames    except OSError:        print 'No Such Directory: %s, exit.' % dir_path        os._exit(1)  def batchrename(dir_path, prefix=DEFAULT_PREFIX ,generator_func=number_generator()):    '''    rename files (such as xxx.[jpg, png, etc]) in the directory specified by dir_path to [prefix][designator].[jpg, png, etc], designator is generated by generator_fuc    '''    names = getDirFiles(dir_path)    for filename in names:        old_filename = PathUtil.join(dir_path,filename)        if PathUtil.isfile(old_filename)==True:             newname=prefix.upper() + generator_func() + '.' + getFileSuffix(filename)            os.rename(old_filename,PathUtil.join(dir_path,newname))            print '%s rename to %s.' % (filename, newname)  # should be info log        else:            batchrename(dir_path+'/'+filename, prefix, generator_func)def getFileSuffix(filename):    try:        sep_ind = filename.index('.')        return filename[sep_ind+1:]    except ValueError:        return Nonedef testGetFileSuffix():    assert getFileSuffix("good.jpg") == "jpg"    assert getFileSuffix("good") is None    print "testGetFileSuffix Passed."def testNumberGenerator():    geneNums = []    generator = number_generator()    for i in range(10):        geneNums.append(generator())     assert geneNums[0] == '0001'    assert geneNums[1] == '0002'    assert geneNums[9] == '0010'    print 'testNumberGenerator Passed.'if __name__ == '__main__':    testGetFileSuffix()    testNumberGenerator()        (dir_path_list, prefix_list, (method, start_num, bits)) = parseArgs()        dir_num = len(dir_path_list)    for i in range(dir_num):        if method == NUM_METHOD:            number_generator_func = number_generator(start_num, bits)        batchrename(dir_path_list[i], prefix_list[i], number_generator_func)

 

Performance cost

Programmers have obsessive-compulsive disorder in pursuit of efficiency. Imagine this is a web Service. The performance cost is usually reflected in the response speed and throughput. The response speed is perceptible to users and affects user experience. The throughput is imperceptible to users and affects service costs. In this example, you can consider renaming millions of files. There are two factors that affect efficiency: 1. File Name sorting time; 2. rename system call time. For the former, use quick sorting, or use more elaborate methods to solve the OS in the batchrename function. by default, rename overwrites existing files (this reduces maintainability). For the latter, if the programming platform or System Call provides a more efficient batch rename interface, you can call this interface to complete tasks in batches.

　

　　Conclusion

Improving program quality is not achieved overnight, but can be achieved in a gradual manner. When a basic available program is implemented, it is still at the starting point. You need to ask yourself:

1. Robustness: What kind of running environment and input parameters does the program need? What should the program do if the running environment is not met or the input parameters are invalid?

2. customization: What parameters or features of a program can be customized? Do not write in the code;

3. Traceability: What are the key running states and key running paths of the program? Use info to log down;

4. Security: under what circumstances may a program damage user data? How does a program prohibit illegal programs from destroying or spying on user data?

5. Scalability: What are the potential and reasonable requirements for changes in the program?

6. reusability: Is the function method bloated and can be used to extract reusable sub-processes?

7. testability: Are there sufficient unit tests for key functions and methods?

8. performance cost: Is the response speed within the user's acceptance range? Can I optimize the local environment without reducing the maintainability and improve the overall throughput? Can programs cope with large data volumes? What is the maximum throughput of a program?

 

　　Indicate the source for reprinting. Thanks :)