pyutilib.autotest A framework for generating test suites from external configuration files
Download

pyutilib.autotest Ranking & Summary

pyutilib.autotest Tags

Python Python framework testing test suites suites

pyutilib.autotest Description

A framework for generating test suites from external configuration files pyutilib.autotest is a package that provides a facility for automatically configuring tests that are executed with Python's unittest package. This capability is tailored for tests where one or more solvers are applied to one or more problems. This testing structure is particularly useful for evaluating the execution of external executables on datasets.There are three main steps for configuring and applying pyutilib.autotest: 1. Create a configuration file 2. Create a test driver plugin 3. Setup the Python module that will be used to apply the testsThese steps are described in the following subsections.Creating a Configuration FileCurrently, pyutilib.autotest only supports a YAML configuration file. The top-level collection in this configuration is a mapping with the following keys:python This section contains a list of Python expressions that are executed while setting up the test suites.driver This section specifies the name of the test driver that is used to execute tests.solvers This section contains a mapping of solver names to solver options.problems This section contains a mapping of problem names to problem options.suites This section contains a mapping of test suite names to suite configurations.The following example illustrates the structure of a YAML configuration file. (This example is the file pyutilib.autotest/examples/autotest/example1.yml in the pyutilib.autotest distribution.)driver: examplepython: - import examplesolvers: cat: cat2: name: cat cat_options: -n echo:problems: p1: file: README.txt p2: file: example.py p3: file: LoremIpsum1.txtsuites: suite1: categories: - smoke - suite1 solvers: cat: cat2: echo: problems: p1: suite2: categories: - nightly - suite2 solvers: cat: cat2: echo: problems: p1: p2: p3: tests: - solver: cat problem: p1 - solver: cat2 problem: p2 - solver: echo problem: p3 suite3: categories: - suite3 solvers: cat: catx: solver: cat cat_options: -n problems: p1:The test driver example is defined in the example.py file, which is imported with the directives in the python section.Within the solvers and problems sections, each solver and problem can specify additional options that are passed into the test. Note that these options are not distinguished for the test; option name conflicts will results in unpredictable behavior.Three solvers are defined in this example, which apply the unix cat and echo commands. By default, the name of the solver is assumed to be the name of the key for the solver map. Solver cat2 illustrates how a solver name can be specified separately from the solver key.The suites section defines one or more test suites. Each test suite consists of a mapping with the following sections:categories This optional section contains a list of strings that are solver categories. These categories can be used to select characteristics of the test suites that are executed.solvers This section contains a mapping of solvers. Note that additional options can be provided here to further customize the solver behavior. By default, the solver name is assumed to be the key value, but the solver key can be used to explicitly define the solver that is associated with a customized solver mapping.problems This section contains a list of problems. Note that additional options can be provided here to further customize the test behavior. By default, the problem name is assumed to be the key value, but the problem key can be used to explicitly define the problem that is associated with a customized problem mapping.tests This optional section contains a list of solver-problem pairs, which defines the actual tests that are defined. Note that the solver and problem values map to the keys defined in the solvers and problems sections of a test suite; thus, this section can use the names of customized solvers and problems that are not defined in the top-level solvers and problems sections. Finally, if this section is omitted, then all combinations of solvers and problems are used to create tests.In this example, three suites are defined to illustrate different features of the test driver:suite1 A simple test suite in which all solvers are applied to a single problem.suite2 The tests section is specified to select a subset of all combinations of solvers and problems to test.suite3 The catx solver is customized from cat to operate like the cat2 solver.Creating a Test DriverThe test configuration used by pyutilib.autotest is quite generic. It specifies what combinations of solvers and problems are to be tested, along with their corresponding options. However, it does not specify how the test is performed. This is done by a test driver class.In general, test driver classes are required to be plugins that can be dynamically created by pyutilib.autotest to execute tests. The easiest way to define a test driver plugin is to inherit from the TestDriverBase class. For example, the following test driver is used in by the earlier test configuration; this plugin is defined in pyutilib.autotest/examples/autotest/example.py.import pyutilib.autotestfrom pyutilib.component.core import *import pyutilib.subprocessclass ExampleTestDriver(pyutilib.autotest.TestDriverBase): """ This test driver executes a unix command and compares its output with a baseline value. """ alias('example') def run_test(self, testcase, name, options): """Execute a single test in the suite""" name = options.suite+'_'+name cmd = options.solver+' ' if not options.cat_options is None: cmd += options.cat_options+' ' cmd += options.file print "Running test suite '%s' test '%s' command '%s'" % \ (options.suite, name, cmd) pyutilib.subprocess.run(cmd, outfile=options.currdir+'test_'+name+".out") testcase.failUnlessFileEqualsBaseline( options.currdir+'test_'+name+".out", options.currdir+'test_'+name+".txt")The alias function is used to specify the name of this plugin; this is the name of the test driver used in the test configuration file.The run_test method executes a single test in the test suite. Note that this method is passed in testcase, which is the test suite class. Thus, this method can directly apply the unittest methods that are defined in this class (e.g. assertEquals).In this example, a unix command-line is create from the solver name, the solver options, and the problem filename. This is executed with the pyutilib.subprocess.run function, which redirects output to a log file. This log file is then compared with a baseline file using the failUnlessFileEqualsBaseline, which is defined in the unittest extensions in pyutilib.th.Note that a variety of other standard unit test methods can also be defined by this test driver. This driver is a ITestDriver plugin, and the API for this plugin is:class ITestDriver(Interface): def setUpClass(self, cls, options): """Set-up the class that defines the suite of tests""" def tearDownClass(self, cls, options): """Tear-down the class that defines the suite of tests""" def setUp(self, testcase, options): """Set-up a single test in the suite""" def tearDown(self, testcase, options): """Tear-down a single test in the suite""" def run_test(self, testcase, name, options): """Execute a single test in the suite"""Creating a Test ModuleVirtually all of the work needed to create test suites is automated by pyutilib.autotest. The following test module is used in this example; (see pyutilib.autotest/examples/autotest/autotest.py):import osimport sysfrom os.path import abspath, dirnamecurrdir = dirname(abspath(__file__))+os.sepimport pyutilib.th as unittestimport pyutilib.autotestif __name__ == "__main__": pyutilib.autotest.create_test_suites(filename=currdir+'example1.yml', _globals=globals()) unittest.main()The first four lines are needed to identify the current directory, where the test configuration file resides.Note that pyutilib.th is imported as unittest, which reminds the user that this is a unittest extension package. (Specifically, this package contains hooks needed to dynamically add functions as test methods in test suites.)The pyutilib.autotest packages is imported so the create_test_suites function can be executed. The arguments to this function are the test configuration file, and the global dictionary.Finally, unittest.main() is executed, as in any unittest module. Tests can be executed using standard unittest command-line options. One extension to this behavior is the use of the PYUTILIB_AUTOTEST_CATEGORIES or PYUTILIB_UNITTEST_CATEGORIES environmental variables; if one of these is specified, then pyutilib.autotest assumes that this data contains a comma-separated list of categories that are used to select the test suites that are constructed. Specifically, if a test suite contains one of the specified test categories, then it will be executed.The pyutilib_test_driver CommandThe pyutilib_test_driver command can be used to execute tests defined in a configuration file without creating a test module. In practice, test modules are typically needed to support test discovery with tools like nose. However, this command provides several features that are useful when diagnosing tests.The command-line behavior of pyutilib_test_driver extends the API of unittest.main(). The following additional options are provided to allow the user to interrogate the tests that are defined by the test configuration file:--help-suites Print the test suites that can be executed--help-tests=HELP_TESTS Print the tests in the specified test suite--help-categories Print the test suite categories that can be specified Requirements: · Python

pyutilib.autotest Related Software