2016-01-28 15:56:10 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (C) 2016-2018 Edward d'Auvergne
|
|
|
|
|
*
|
|
|
|
|
* This file is part of the program FlightGear.
|
|
|
|
|
*
|
|
|
|
|
* This program is free software: you can redistribute it and/or modify
|
|
|
|
|
* it under the terms of the GNU General Public License as published by
|
|
|
|
|
* the Free Software Foundation, either version 2 of the License, or
|
|
|
|
|
* (at your option) any later version.
|
|
|
|
|
*
|
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
|
*
|
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
|
* along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#include <cppunit/TestResult.h>
|
|
|
|
|
#include <cppunit/TestResultCollector.h>
|
|
|
|
|
#include <cppunit/TextTestRunner.h>
|
|
|
|
|
#include <cppunit/extensions/TestFactoryRegistry.h>
|
|
|
|
|
|
TestSuite: Custom output formatting.
A large number of improvements have been made by subclassing the CppUnit classes
and overriding the base functions, including CompilerOutputter, TestListener,
and TextTestRunner. All IO has also been captured by the test runner.
The result is that during the test suite run, only the characters '.', 'F', and
'E' are output for a pass, failure and error state. At the end of each test
suite category, all failures and errors are reported in full detail, including
the different captured IO streams. A final synopsis is printed out, improving
the overview in the case of too many tests failing.
For the fgCompilerOutputter class, the printSuccess(), printFailureReport(),
printFailureDetail(), and printSuiteStats() functions have been replaced to
implement the above printout design. The class also stores the std::vector of
TestIOCapt structures for the final printouts.
The fgTestListener class handles the events from the running of the test suite.
The startTest() and endTest() functions are used for IO capture. The IO is
placed into a TestIOCapt data structure, with one std::string for holding the
combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures
occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector.
The startTest() and endTest() functions are also used for starting and stopping
a timer to allow the full test suite to be timed. And the addFailure() function
simply registers test failures or errors.
The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function,
simply to prevent the base method from spawning a second test listener, causing
the test output to be duplicated.
Some auxiliary formatting functions have been added to print out titles,
sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
|
|
|
#include "fgCompilerOutputter.hxx"
|
|
|
|
|
#include "fgTestListener.hxx"
|
|
|
|
|
#include "formatting.hxx"
|
|
|
|
|
|
2016-01-28 15:56:10 +00:00
|
|
|
|
|
|
|
|
// Execute all test suites for the given test category.
|
2018-06-20 09:29:29 +00:00
|
|
|
int testRunner(const std::string& type, const std::string& title, char *subset, bool verbose, bool ctest_output, bool debug)
|
2016-01-28 15:56:10 +00:00
|
|
|
{
|
|
|
|
|
// Declarations.
|
|
|
|
|
CppUnit::TextTestRunner runner;
|
|
|
|
|
|
TestSuite: Custom output formatting.
A large number of improvements have been made by subclassing the CppUnit classes
and overriding the base functions, including CompilerOutputter, TestListener,
and TextTestRunner. All IO has also been captured by the test runner.
The result is that during the test suite run, only the characters '.', 'F', and
'E' are output for a pass, failure and error state. At the end of each test
suite category, all failures and errors are reported in full detail, including
the different captured IO streams. A final synopsis is printed out, improving
the overview in the case of too many tests failing.
For the fgCompilerOutputter class, the printSuccess(), printFailureReport(),
printFailureDetail(), and printSuiteStats() functions have been replaced to
implement the above printout design. The class also stores the std::vector of
TestIOCapt structures for the final printouts.
The fgTestListener class handles the events from the running of the test suite.
The startTest() and endTest() functions are used for IO capture. The IO is
placed into a TestIOCapt data structure, with one std::string for holding the
combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures
occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector.
The startTest() and endTest() functions are also used for starting and stopping
a timer to allow the full test suite to be timed. And the addFailure() function
simply registers test failures or errors.
The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function,
simply to prevent the base method from spawning a second test listener, causing
the test output to be duplicated.
Some auxiliary formatting functions have been added to print out titles,
sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
|
|
|
// Print out a title.
|
2018-03-12 10:43:24 +00:00
|
|
|
if (!ctest_output)
|
|
|
|
|
printTitle(std::cerr, title);
|
TestSuite: Custom output formatting.
A large number of improvements have been made by subclassing the CppUnit classes
and overriding the base functions, including CompilerOutputter, TestListener,
and TextTestRunner. All IO has also been captured by the test runner.
The result is that during the test suite run, only the characters '.', 'F', and
'E' are output for a pass, failure and error state. At the end of each test
suite category, all failures and errors are reported in full detail, including
the different captured IO streams. A final synopsis is printed out, improving
the overview in the case of too many tests failing.
For the fgCompilerOutputter class, the printSuccess(), printFailureReport(),
printFailureDetail(), and printSuiteStats() functions have been replaced to
implement the above printout design. The class also stores the std::vector of
TestIOCapt structures for the final printouts.
The fgTestListener class handles the events from the running of the test suite.
The startTest() and endTest() functions are used for IO capture. The IO is
placed into a TestIOCapt data structure, with one std::string for holding the
combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures
occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector.
The startTest() and endTest() functions are also used for starting and stopping
a timer to allow the full test suite to be timed. And the addFailure() function
simply registers test failures or errors.
The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function,
simply to prevent the base method from spawning a second test listener, causing
the test output to be duplicated.
Some auxiliary formatting functions have been added to print out titles,
sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
|
|
|
|
2016-01-28 15:56:10 +00:00
|
|
|
// Get all tests.
|
2018-06-20 09:29:29 +00:00
|
|
|
runner.addTest(CppUnit::TestFactoryRegistry::getRegistry(type).makeTest());
|
2016-01-28 15:56:10 +00:00
|
|
|
|
TestSuite: Custom output formatting.
A large number of improvements have been made by subclassing the CppUnit classes
and overriding the base functions, including CompilerOutputter, TestListener,
and TextTestRunner. All IO has also been captured by the test runner.
The result is that during the test suite run, only the characters '.', 'F', and
'E' are output for a pass, failure and error state. At the end of each test
suite category, all failures and errors are reported in full detail, including
the different captured IO streams. A final synopsis is printed out, improving
the overview in the case of too many tests failing.
For the fgCompilerOutputter class, the printSuccess(), printFailureReport(),
printFailureDetail(), and printSuiteStats() functions have been replaced to
implement the above printout design. The class also stores the std::vector of
TestIOCapt structures for the final printouts.
The fgTestListener class handles the events from the running of the test suite.
The startTest() and endTest() functions are used for IO capture. The IO is
placed into a TestIOCapt data structure, with one std::string for holding the
combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures
occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector.
The startTest() and endTest() functions are also used for starting and stopping
a timer to allow the full test suite to be timed. And the addFailure() function
simply registers test failures or errors.
The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function,
simply to prevent the base method from spawning a second test listener, causing
the test output to be duplicated.
Some auxiliary formatting functions have been added to print out titles,
sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
|
|
|
// Set up the test listener.
|
|
|
|
|
fgTestListener *testListener;
|
|
|
|
|
testListener = new fgTestListener;
|
|
|
|
|
runner.eventManager().addListener(testListener);
|
TestSuite: Addition of command line options for fine control of the executable.
This includes the following help message detailing all of the new options:
"""
Usage: run_test_suite [options]
Options:
-h, --help show this help message and exit.
Test selection options:
-s, --system-tests execute the system/functional tests.
-u, --unit-tests execute the unit tests.
-g, --gui-tests execute the GUI tests.
-m, --simgear-tests execute the simgear tests.
The -s, -u, -g, and -m options accept an optional argument to perform a
subset of all tests. This argument should either be the name of a test
suite or the full name of an individual test.
Full test names consist of the test suite name, the separator '::' and then
the individual test name. The test names can revealed with the verbose
option.
Verbosity options:
-v, --verbose verbose output including names and timings for all
tests.
"""
2018-03-09 23:02:14 +00:00
|
|
|
testListener->verbose = verbose;
|
2018-03-12 10:43:24 +00:00
|
|
|
testListener->ctest_output = ctest_output;
|
2018-03-12 15:43:24 +00:00
|
|
|
testListener->debug = debug;
|
TestSuite: Custom output formatting.
A large number of improvements have been made by subclassing the CppUnit classes
and overriding the base functions, including CompilerOutputter, TestListener,
and TextTestRunner. All IO has also been captured by the test runner.
The result is that during the test suite run, only the characters '.', 'F', and
'E' are output for a pass, failure and error state. At the end of each test
suite category, all failures and errors are reported in full detail, including
the different captured IO streams. A final synopsis is printed out, improving
the overview in the case of too many tests failing.
For the fgCompilerOutputter class, the printSuccess(), printFailureReport(),
printFailureDetail(), and printSuiteStats() functions have been replaced to
implement the above printout design. The class also stores the std::vector of
TestIOCapt structures for the final printouts.
The fgTestListener class handles the events from the running of the test suite.
The startTest() and endTest() functions are used for IO capture. The IO is
placed into a TestIOCapt data structure, with one std::string for holding the
combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures
occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector.
The startTest() and endTest() functions are also used for starting and stopping
a timer to allow the full test suite to be timed. And the addFailure() function
simply registers test failures or errors.
The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function,
simply to prevent the base method from spawning a second test listener, causing
the test output to be duplicated.
Some auxiliary formatting functions have been added to print out titles,
sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
|
|
|
|
2016-01-28 15:56:10 +00:00
|
|
|
// Set the test suite output IO stream.
|
2018-03-12 15:43:24 +00:00
|
|
|
runner.setOutputter(new fgCompilerOutputter(&runner.result(), &testListener->io_capt, &testListener->sum_time, std::cerr, ctest_output, debug));
|
2016-01-28 15:56:10 +00:00
|
|
|
|
|
|
|
|
// Execute the tests.
|
TestSuite: Addition of command line options for fine control of the executable.
This includes the following help message detailing all of the new options:
"""
Usage: run_test_suite [options]
Options:
-h, --help show this help message and exit.
Test selection options:
-s, --system-tests execute the system/functional tests.
-u, --unit-tests execute the unit tests.
-g, --gui-tests execute the GUI tests.
-m, --simgear-tests execute the simgear tests.
The -s, -u, -g, and -m options accept an optional argument to perform a
subset of all tests. This argument should either be the name of a test
suite or the full name of an individual test.
Full test names consist of the test suite name, the separator '::' and then
the individual test name. The test names can revealed with the verbose
option.
Verbosity options:
-v, --verbose verbose output including names and timings for all
tests.
"""
2018-03-09 23:02:14 +00:00
|
|
|
if (subset == NULL)
|
|
|
|
|
runner.run("", false, true, false);
|
|
|
|
|
else
|
|
|
|
|
runner.run(subset, false, true, false);
|
2016-01-28 15:56:10 +00:00
|
|
|
|
TestSuite: Custom output formatting.
A large number of improvements have been made by subclassing the CppUnit classes
and overriding the base functions, including CompilerOutputter, TestListener,
and TextTestRunner. All IO has also been captured by the test runner.
The result is that during the test suite run, only the characters '.', 'F', and
'E' are output for a pass, failure and error state. At the end of each test
suite category, all failures and errors are reported in full detail, including
the different captured IO streams. A final synopsis is printed out, improving
the overview in the case of too many tests failing.
For the fgCompilerOutputter class, the printSuccess(), printFailureReport(),
printFailureDetail(), and printSuiteStats() functions have been replaced to
implement the above printout design. The class also stores the std::vector of
TestIOCapt structures for the final printouts.
The fgTestListener class handles the events from the running of the test suite.
The startTest() and endTest() functions are used for IO capture. The IO is
placed into a TestIOCapt data structure, with one std::string for holding the
combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures
occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector.
The startTest() and endTest() functions are also used for starting and stopping
a timer to allow the full test suite to be timed. And the addFailure() function
simply registers test failures or errors.
The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function,
simply to prevent the base method from spawning a second test listener, causing
the test output to be duplicated.
Some auxiliary formatting functions have been added to print out titles,
sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
|
|
|
// Clean up.
|
|
|
|
|
delete testListener;
|
|
|
|
|
|
2016-01-28 15:56:10 +00:00
|
|
|
// Return the status of the tests.
|
|
|
|
|
CppUnit::TestResultCollector &status = runner.result();
|
2018-05-07 10:41:22 +00:00
|
|
|
return status.testFailuresTotal();
|
2016-01-28 15:56:10 +00:00
|
|
|
}
|