Use the unittest and doctest modules in Python, unittestdoctest

Use the unittest and doctest modules in Python, unittestdoctest

I want to be honest. Although I am the creator of a Python library in a widely used public domain, the unit tests introduced in my modules are very unsystematic. In fact, most of those tests are included in gnosis Utilities of Gnosis. xml. pickle and are written by contributors to this sub-package. I also found that the vast majority of third-party Python packages I downloaded lacked a complete unit test set.

In addition, existing tests in Gnosis Utilities are also trapped by another defect: You often need to deduce the expected output in a large amount of detail to determine the success or failure of the test. The test is actually-in many cases-more like a small utility that uses some parts of the library. These tests (or utilities) Support input and/or descriptive data format output from any data source (of the correct type. In fact, these test utilities are more useful when you need to debug minor errors. However, for the sanity checks of self-explanatory integrity checks for database version changes, these class tests are not competent.

In this article, I tried to use the Python standard library modules doctest and unittest to improve the centralized test of my utility, and lead you to experience with me (and point out some of the best methods ).

The script gnosis/xml/objectid/test/test_basic.py provides a typical example of the shortcomings of the current test and the solution. The latest version of the script is as follows:

Listing 1. test_basic.py

"Read and print and objectified XML file"import sysfrom gnosis.xml.objectify import XML_Objectify, pyobj_printerif len(sys.argv) > 1: for filename in sys.argv[1:]:  for parser in ('DOM','EXPAT'):   try:    xml_obj = XML_Objectify(filename, parser=parser)    py_obj = xml_obj.make_instance()    print pyobj_printer(py_obj).encode('UTF-8')    sys.stderr.write("++ SUCCESS (using "+parser+")\n")    print "="*50   except:    sys.stderr.write("++ FAILED (using "+parser+")\n")    print "="*50else: print "Please specify one or more XML files to Objectify."

The utility function pyobj_printer () generates any Python object (such an object is not used in gnosis. xml. any other utility of objectid does not use a non-XML Representation of anything in Gnosis Utilities. In future versions, I may move this function to other places in the Gnosis package. In any case, pyobj_printer () uses various types-indentation and symbols of Python to describe objects and their attributes (similar to pprint, but extends the instance, not limited to extended built-in data types ).

If some special XML may not be "objectified" correctly, the test_basic.py script provides a good debugging tool-you can visually view the attributes and values of the result object. In addition, if you have redirected STDOUT, you can view simple messages on STDERR, as shown in the following example:

Listing 2. Analyze STDERR result messages

$ python test_basic.py testns.xml > /dev/null++ SUCCESS (using DOM)++ FAILED (using EXPAT)

However, in the preceding running example, the definition of "success" or "failure" is not obvious: "success" only means that no exception occurs, but does not mean that the (redirected) output is correct.
Use doctest

The doctest module allows you to embed comments in docstrings to display the expected behaviors of various statements, especially the results of functions and methods. This is like making the document string look like an interactive shell session. A simple method to complete this task is, copy and paste from a Python Interactive shell (or from Idel, PythonWin, MacPython, or other IDE with interactive sessions. This improved test_basic.py script illustrates how to add a self-diagnosis function:

Listing 3. test_basic.py script with self-diagnosis function

import sysfrom gnosis.xml.objectify import XML_Objectify, pyobj_printer, EXPAT, DOMLF = "\n"def show(xml_src, parser): """Self test using simple or user-specified XML data >>> xml = '''<?xml version="1.0"?> ... <!DOCTYPE Spam SYSTEM "spam.dtd" > ... <Spam> ... <Eggs>Some text about eggs.</Eggs> ... <MoreSpam>Ode to Spam</MoreSpam> ... </Spam>''' >>> squeeze = lambda s: s.replace(LF*2,LF).strip() >>> print squeeze(show(xml,DOM)[0]) -----* _XO_Spam *----- {Eggs}  PCDATA=Some text about eggs. {MoreSpam}  PCDATA=Ode to Spam >>> print squeeze(show(xml,EXPAT)[0]) -----* _XO_Spam *----- {Eggs}  PCDATA=Some text about eggs. {MoreSpam}  PCDATA=Ode to Spam PCDATA= """ try:  xml_obj = XML_Objectify(xml_src, parser=parser)  py_obj = xml_obj.make_instance()  return (pyobj_printer(py_obj).encode('UTF-8'),    "++ SUCCESS (using "+parser+")\n") except:  return ("","++ FAILED (using "+parser+")\n")if __name__ == "__main__": if len(sys.argv)==1 or sys.argv[1]=="-v":  import doctest, test_basic  doctest.testmod(test_basic) elif sys.argv[1] in ('-h','-help','--help'):  print "You may specify XML files to objectify instead of self-test"  print "(Use '-v' for verbose output, otherwise no message means success)" else:  for filename in sys.argv[1:]:   for parser in (DOM, EXPAT):    output, message = show(filename, parser)    print output    sys.stderr.write(message)    print "="*50

Note that I put the main code block in the improved (and extended) test script, so that if you specify an XML file in the command line, the script will continue to execute the previous behavior. In this way, you can continue to analyze XML other than test cases and focus only on the Results-or find out the errors in what gnosis. xml. objectid is doing, or just understand its purpose. In the standard mode, you can use the-h or -- help parameters to get instructions on usage.

When test_basic.py is run without any parameters (or the-v parameter used only by doctest), interesting new functions are found. In this example, we run doctest on the module/script itself -- you can see that we actually import test_basic to the script's own namespace, in this way, we can simply import other modules to be tested. Doctest. the testmod () function traverses the module itself, its functions, and all the document strings in its class to find all content similar to Interactive shell sessions. In this example, this session is found in the show () function.

The show () Document string illustrates several small "traps (gotchas)" in the design of a doctest session )". Unfortunately, doctest processes empty rows as the end of the session when parsing an explicit session -- therefore, output like pyobj_printer () needs some protection (be munged slightly) for testing. The simplest way is to use a function like squeeze () defined by the document string itself (it just removes the line feed that follows ). In addition, because the document string is a string replacement (escape) after all, the \ n sequence is extended, which makes it a little confusing to change the line feed in the code example. You can use \ n, but I found that the definition of LF solves these problems.

The custom test defined in the show () Document string does not only ensure that no exception occurs (compared with the initial test script ). Check at least one simple XML document for correct objecication ication. Of course, it is still possible that some other XML documents cannot be correctly processed-for example, the namespace XML document testns. xml we tried above encountered an EXPAT parser failure. Document strings processed by doctest may contain traceback internally, but in special cases, it is better to use unittest.

Use unittest

Another test included in gnosis. xml. objectid is test_expat.py. The main reason for creating this test is, subsoftware packages using the EXPAT parser users often need to call a special setting function to enable processing of XML documents with namespaces (this is actually evolved rather than designed, and may change in the future ). The old test will try to print the object without the help of settings. If an exception occurs, it will be captured, and then print it with the help of settings if necessary (and give a message about what happened ).

If you use the test_basic.py and test_expat.py tools, you can analyze how gnosis. xml. objectid describes a novel XML document. However, as before, there are many specific behaviors that we may want to verify. An enhanced, extended version of test_expat.py uses unittest to analyze events that occur when various actions are executed, including assertions that hold specific conditions or (approximation) equations, or some expected exceptions. Take a look:

Listing 4. self-diagnosed test_expat.py script

"Objectify using Expat parser, namespace setup where needed"import unittest, sys, cStringIOfrom os.path import isfilefrom gnosis.xml.objectify import make_instance, config_nspace_sep,\         XML_ObjectifyBASIC, NS = 'test.xml','testns.xml'class Prerequisite(unittest.TestCase): def testHaveLibrary(self):  "Import the gnosis.xml.objectify library"  import gnosis.xml.objectify def testHaveFiles(self):  "Check for sample XML files, NS and BASIC"  self.failUnless(isfile(BASIC))  self.failUnless(isfile(NS))class ExpatTest(unittest.TestCase): def setUp(self):  self.orig_nspace = XML_Objectify.expat_kwargs.get('nspace_sep','') def testNoNamespace(self):  "Objectify namespace-free XML document"  o = make_instance(BASIC) def testNamespaceFailure(self):  "Raise SyntaxError on non-setup namespace XML"  self.assertRaises(SyntaxError, make_instance, NS) def testNamespaceSuccess(self):  "Sucessfully objectify NS after setup"  config_nspace_sep(None)  o = make_instance(NS) def testNspaceBasic(self):  "Successfully objectify BASIC despite extra setup"  config_nspace_sep(None)  o = make_instance(BASIC) def tearDown(self):  XML_Objectify.expat_kwargs['nspace_sep'] = self.orig_nspaceif __name__ == '__main__': if len(sys.argv) == 1:  unittest.main() elif sys.argv[1] in ('-q','--quiet'):  suite = unittest.TestSuite()  suite.addTest(unittest.makeSuite(Prerequisite))  suite.addTest(unittest.makeSuite(ExpatTest))  out = cStringIO.StringIO()  results = unittest.TextTestRunner(stream=out).run(suite)  if not results.wasSuccessful():   for failure in results.failures:    print "FAIL:", failure[0]   for error in results.errors:    print "ERROR:", error[0] elif sys.argv[1].startswith('-'): # pass args to unittest  unittest.main() else:  from gnosis.xml.objectify import pyobj_printer as show  config_nspace_sep(None)  for fname in sys.argv[1:]:   print show(make_instance(fname)).encode('UTF-8')

Using unittest adds a lot of capabilities to the simpler doctest method. We can divide our tests into several classes, each of which inherits from unittest. TestCase. In each test class, each method whose name starts with ". test" is considered another test. The two additional classes defined for ExpatTest are interesting: Run before each test of the class used. setUp (), run at the end of the test. tearDown () (whether the test is successful, failed, or error ). In our example above, we made a little bookkeeping for the dedicated expat_kwargs dictionary to ensure that each test runs independently.

By the way, the difference between failure and error is very important. A test may fail because some specific assertions are invalid (the asserted method may start with ". fail" or start with ". assert ). In a sense, failure is expected-at least in a sense, we have analyzed it in detail. On the other hand, errors are unexpected-because we do not know where errors will occur in advance, we need to analyze the backtracking in the actual test run to diagnose such problems. However, we can design a prompt for failure diagnosis errors. For example, if Prerequisite. haveFiles () fails, an error will occur in some TestExpat tests. If the former is successful, you will have to go elsewhere to find the root cause of the error.

In unittest. in the inheritance class of TestCase, the specific test method may include some. assert... () or. fail... () method, but it may only have a series of actions that we believe should be successfully executed. If the test method is not run as expected, we will get an error (and the rollback that describes this error ).

The _ main _ block in test_expat.py is also worth checking. In the simplest case, we can only use unittest. main () to run the test case, which will determine what needs to be run. In this way, the unittest module accepts a-v option to provide more detailed output. Based on the specified file name, after the namespace settings are executed, we print the representation of the specified XML file, thus maintaining backward compatibility with earlier versions of this tool.

The most interesting branch in _ main _ is the branch that expects the-q or -- quiet tag. If you expect, this branch will be silent unless a failure or error occurs (quiet, that is, minimize the output ). Not only that, because it is silent, it only displays a row of reports about the failure/error location for each problem, rather than the whole diagnosis backtracking. In addition to the direct use of the silent output style, this branch also provides examples of custom tests relative to the test suite and control of the results report. The default output of a slightly longer unittest. TextTestRunner () is directed to StringIO out. If you want to view it, you are welcome to go to out. getvalue. However, the result object allows us to test the overall success. If it is not completely successful, we can also handle failures and errors. Obviously, because they are values in the variable, you can easily record the content of the result object into the log, or display it in the GUI, no matter what, not just print it to STDOUT.

Combination test

Probably the best feature of the unittest framework is that you can easily combine tests that contain different modules. In fact, if you use Python 2.3 +, you can even convert the doctest test to the unittest suite. Let's combine the tests we have created so far into a script test_all.py (it is indeed an exaggeration to say it is the test we have done so far ):

Listing 5. test_all.py combines unit test

"Combine tests for gnosis.xml.objectify package (req 2.3+)"import unittest, doctest, test_basic, test_expatsuite = doctest.DocTestSuite(test_basic)suite.addTest(unittest.makeSuite(test_expat.Prerequisite))suite.addTest(unittest.makeSuite(test_expat.ExpatTest)) unittest.TextTestRunner(verbosity=2).run(suite)

Because test_expat.py only contains test classes, they can be easily added to the local test suite. The doctest. DocTestSuite () function executes the conversion of the document string test. Let's take a look at what test_all.py will do during runtime:

Listing 6. Successful output from test_all.py

$ python2.3 test_all.pydoctest of test_basic.show ... okCheck for sample XML files, NS and BASIC ... okImport the gnosis.xml.objectify library ... okRaise SyntaxError on non-setup namespace XML ... okSucessfully objectify NS after setup ... okObjectify namespace-free XML document ... okSuccessfully objectify BASIC despite extra setup ... ok----------------------------------------------------------------------Ran 7 tests in 0.052sOK

Note the description of the test to be executed: When the unittest method is used, their description comes from the corresponding docstring function. If you do not specify a document string, the class and method names are used as the most appropriate descriptions. Let's take a look at what we will get when some tests fail. It's also interesting (we removed the tracing details for this article ):

Listing 7. Results when some tests fail

$ mv testns.xml testns.xml# && python2.3 test_all.py 2>&1 | head -7doctest of test_basic.show ... okCheck for sample XML files, NS and BASIC ... FAILImport the gnosis.xml.objectify library ... okRaise SyntaxError on non-setup namespace XML ... ERRORSucessfully objectify NS after setup ... ERRORObjectify namespace-free XML document ... okSuccessfully objectify BASIC despite extra setup ... ok

As mentioned above, the last line of this failure written to STDERR is "FAILED (failures = 1, errors = 2 )", if you need it, this is a good summary (relative to the "OK" in the end when the success is successful ").

Start from here

This article introduces some typical usage of unittest and doctest, which have improved testing in my own software. Read the Python documentation to learn more about the full range of methods that can be used in Test suites, test cases, and test results. All of them follow the pattern described in the example.

It is a good software practice to bring yourself into compliance with the methodology defined by the Python standard test module. The development of test-driven is popular in many software cycles. However, it is clear that Python is a language suitable for testing driver models. Moreover, if you only consider a software package, it is more likely to work as planned. If a software package or library is accompanied by a comprehensive set of tests, it will be more useful to users than the software packages or libraries without these tests.