Regression testing in the toolkit

Note: testsuite has been moved to https://github.com/dita-ot/testsuite

Regression testing with the toolkit

Note: This information is primarily of interest to toolkit developers and those creating plugins for the toolkit. For information on the toolkit itself, see The DITA Open Toolkit.

The core toolkit development team is working on the release of its regression testing package. The package consists of a large number of sample DITA maps and topics representing a wide variety of content, as well as batch files to run the content through a build. The regression test package will be posted to SourceForge, and this page will be updated with documentation on how to run the test.

The test is generally used by running all of the builds to generate output after making a modification to the code. The output of the test is compared to the output with the previous, working set of code to determine if any tests fail or if any output changes in a negative way. This is most useful when making changes to the base code or when testing a plugin that overrides core code (including overrides that should affect only a related specialization).

The toolkit development welcomes additions to the regression test bucket, as well as suggestions for ways to improve the testing process.

The regression suite package is committed to CVS. Developers can access the package by exporting it or checking it out from CVS. Exporting action does not need access authority to CVS. Here is the information about the CVS repository.

Connect protocol: ssh

Host: dita-ot.cvs.sourceforge.net

Port: Default

Repository Path: /cvsroot/dita-ot

Module: RegressionSuite

The structure of regression suite package is listed as followings:

testcase: ant scripts to run corresponding test

testdata: input data that is needed by test

*.bat: windows batch file to run the test

testbuild.xml: the stub ant script to start all testing

After regression suite package is downloaded, the content in the package (testcase, testdata, *.bat and testbuild.xml) should be extracted andcopied to installation directory of DITA OT. Then you need to enter the DITA OT installation directory in console and use ant -f testbuild.xml to start the regression test. 

More information will be coming when the test is available at SourceForge.

I don't see any "tests" in this suite (by "test", I mean something that explicitly passes or fails, like a junit or antunit test)?  Currently, this looks like a batch job that you can run before and after changes, and then compare output as you wish.

Why not go a little further and make this a "test suite".  E.g. for each of the roughly 75 test cases, you could have a coresponding AntUnit test. for each of the topic files, you could have an assertFilesMatch assertion (that compared the expected output file, with the file that was produced during that specific run of the suite)

As you increment versions of the toolkit, you can modify the suite accordingly and tag/branch it along with the dita-ot
XML.org Focus Areas: BPEL | DITA | ebXML | IDtrust | OpenDocument | SAML | UBL | UDDI
OASIS sites: OASIS | Cover Pages | XML.org | AMQP | CGM Open | eGov | Emergency | IDtrust | LegalXML | Open CSA | OSLC | WS-I