fly/flightgear
1
0
Fork 0
Code Activity
flightgear/test_suite/fgTestListener.cxx

139 lines
4 KiB
C++
Raw Normal View History

TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
/*
* Copyright (C) 2016 Edward d'Auvergne
*
* This file is part of the program FlightGear.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <cppunit/Test.h>
#include <cppunit/TestFailure.h>
#include <cppunit/TestListener.h>
#include "fgTestListener.hxx"
TestSuite: Capture and reporting of all simgear logstream priorities. Each of the global logstream priorities are captured into its own std::ostringstream stream, all held together in the test suite global _iostreams class instance. This object can be obtained by calling getIOstreams(). The streams are captured using the StreamLogCallback class, which is a simple modification of the simgear FileLogCallback class, registered with the global logstream's addCallback() function. When tests fail, all of contents the different simgear logstreams are now reported. The failure report consists of the following sections: - Failure information. - SG_BULK simgear logstream (all messages). - SG_BULK only simgear logstream. - SG_DEBUG only simgear logstream. - SG_INFO only simgear logstream. - SG_WARN only simgear logstream. - SG_ALERT only simgear logstream. - Combined STDOUT and STDERR streams. Any empty sections, except for SG_BULK, will not be shown.
2016-02-12 18:42:20 +00:00
#include "logging.hxx"
TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
using namespace std;
// Handle failures.
void fgTestListener::addFailure(const CppUnit::TestFailure &failure)
{
m_failure = true;
if (failure.isError())
m_error = true;
}
// Override the base class function to restore IO streams.
void fgTestListener::endTest(CppUnit::Test *test)
{
// Test timing.
sum_time += clock() - m_time;
// Restore the IO streams.
TestSuite: Added a debugging command line option. This is to enable full output from passing tests to help with debugging.
2018-03-12 15:43:24 +00:00
if (!debug) {
cout.rdbuf(orig_cout);
cerr.rdbuf(orig_cerr);
}
// Debugging output.
if (debug)
cerr << string(WIDTH_DIVIDER, '-') << endl;
TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
// Per-test single character status feedback.
if (m_failure)
cerr << (m_error ? "E" : "F");
else
cerr << '.';
TestSuite: Addition of command line options for fine control of the executable. This includes the following help message detailing all of the new options: """ Usage: run_test_suite [options] Options: -h, --help show this help message and exit. Test selection options: -s, --system-tests execute the system/functional tests. -u, --unit-tests execute the unit tests. -g, --gui-tests execute the GUI tests. -m, --simgear-tests execute the simgear tests. The -s, -u, -g, and -m options accept an optional argument to perform a subset of all tests. This argument should either be the name of a test suite or the full name of an individual test. Full test names consist of the test suite name, the separator '::' and then the individual test name. The test names can revealed with the verbose option. Verbosity options: -v, --verbose verbose output including names and timings for all tests. """
2018-03-09 23:02:14 +00:00
TestSuite: Renamed the 'verbose' command line option to 'timings'. This is to better reflect what the option does. The -v or --verbose options are now called -t or --timings respectively.
2019-08-07 08:08:22 +00:00
// Timing output.
if (timings || ctest_output || debug) {
TestSuite: Addition of command line options for fine control of the executable. This includes the following help message detailing all of the new options: """ Usage: run_test_suite [options] Options: -h, --help show this help message and exit. Test selection options: -s, --system-tests execute the system/functional tests. -u, --unit-tests execute the unit tests. -g, --gui-tests execute the GUI tests. -m, --simgear-tests execute the simgear tests. The -s, -u, -g, and -m options accept an optional argument to perform a subset of all tests. This argument should either be the name of a test suite or the full name of an individual test. Full test names consist of the test suite name, the separator '::' and then the individual test name. The test names can revealed with the verbose option. Verbosity options: -v, --verbose verbose output including names and timings for all tests. """
2018-03-09 23:02:14 +00:00
// Test timing.
float time = ((float)(clock()-m_time))/CLOCKS_PER_SEC;
char buffer[100];
if (time > 60.0)
snprintf(buffer, sizeof(buffer), "%10.3f %-3s", time/60, "min");
else if (time > 1.0)
snprintf(buffer, sizeof(buffer), "%10.3f %-3s", time, "s");
else
snprintf(buffer, sizeof(buffer), "%10.3f %-3s", time*1000, "ms");
cerr << buffer;
// Test name.
cerr << " for " << test->getName() << endl;
}
TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
cerr.flush();
// Store the captured IO for any failed tests.
TestSuite: Added a debugging command line option. This is to enable full output from passing tests to help with debugging.
2018-03-12 15:43:24 +00:00
if (m_failure && !debug) {
TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
// Set up the data structure.
TestIOCapt test_io;
test_io.name = test->getName();
// Standard IO.
test_io.stdio = capt.str();
TestSuite: Capture and reporting of all simgear logstream priorities. Each of the global logstream priorities are captured into its own std::ostringstream stream, all held together in the test suite global _iostreams class instance. This object can be obtained by calling getIOstreams(). The streams are captured using the StreamLogCallback class, which is a simple modification of the simgear FileLogCallback class, registered with the global logstream's addCallback() function. When tests fail, all of contents the different simgear logstreams are now reported. The failure report consists of the following sections: - Failure information. - SG_BULK simgear logstream (all messages). - SG_BULK only simgear logstream. - SG_DEBUG only simgear logstream. - SG_INFO only simgear logstream. - SG_WARN only simgear logstream. - SG_ALERT only simgear logstream. - Combined STDOUT and STDERR streams. Any empty sections, except for SG_BULK, will not be shown.
2016-02-12 18:42:20 +00:00
// The simgear logstreams.
capturedIO &obj = getIOstreams();
TestSuite: Addition of the --log-class option for selecting SG_LOG classes. This option mimics the fgfs option of the same name but additionally supports the "particles", "headless", "osg", "undefined", and "all" classes.
2019-07-31 08:35:23 +00:00
test_io.log_class = obj.log_class;
TestSuite: Addition of the --log-level option for setting the logging priority. This option mimics the fgfs option of the same name. However for the test suite option, additionally the SG_POPUP, SG_DEV_WARN and SG_DEV_ALERT priorities are supported. The default test suite output has been modified to only show the interleaved log with the logging priority set to the command line supplied value or defaulting to SG_INFO.
2019-07-30 11:19:23 +00:00
test_io.log_priority = obj.log_priority;
test_io.sg_interleaved = obj.sg_interleaved.str();
TestSuite: Capture and reporting of all simgear logstream priorities. Each of the global logstream priorities are captured into its own std::ostringstream stream, all held together in the test suite global _iostreams class instance. This object can be obtained by calling getIOstreams(). The streams are captured using the StreamLogCallback class, which is a simple modification of the simgear FileLogCallback class, registered with the global logstream's addCallback() function. When tests fail, all of contents the different simgear logstreams are now reported. The failure report consists of the following sections: - Failure information. - SG_BULK simgear logstream (all messages). - SG_BULK only simgear logstream. - SG_DEBUG only simgear logstream. - SG_INFO only simgear logstream. - SG_WARN only simgear logstream. - SG_ALERT only simgear logstream. - Combined STDOUT and STDERR streams. Any empty sections, except for SG_BULK, will not be shown.
2016-02-12 18:42:20 +00:00
test_io.sg_bulk_only = obj.sg_bulk_only.str();
test_io.sg_debug_only = obj.sg_debug_only.str();
test_io.sg_info_only = obj.sg_info_only.str();
test_io.sg_warn_only = obj.sg_warn_only.str();
test_io.sg_alert_only = obj.sg_alert_only.str();
TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
// Add the test's IO to the list.
io_capt.push_back(test_io);
}
}
// Override the base class function to capture IO streams.
void fgTestListener::startTest(CppUnit::Test *test)
{
TestSuite: Added a debugging command line option. This is to enable full output from passing tests to help with debugging.
2018-03-12 15:43:24 +00:00
// IO capture.
if (!debug) {
// Clear the simgear logstream buffers.
capturedIO &obj = getIOstreams();
TestSuite: Addition of the --log-level option for setting the logging priority. This option mimics the fgfs option of the same name. However for the test suite option, additionally the SG_POPUP, SG_DEV_WARN and SG_DEV_ALERT priorities are supported. The default test suite output has been modified to only show the interleaved log with the logging priority set to the command line supplied value or defaulting to SG_INFO.
2019-07-30 11:19:23 +00:00
obj.sg_interleaved.str("");
TestSuite: Added a debugging command line option. This is to enable full output from passing tests to help with debugging.
2018-03-12 15:43:24 +00:00
obj.sg_bulk_only.str("");
obj.sg_debug_only.str("");
obj.sg_info_only.str("");
obj.sg_warn_only.str("");
obj.sg_alert_only.str("");
// Store the original STDOUT and STDERR for restoring later on.
orig_cout = cout.rdbuf();
orig_cerr = cerr.rdbuf();
// Clear the captured stream and then catch stdout and stderr.
capt.str(string());
cout.rdbuf(capt.rdbuf());
cerr.rdbuf(capt.rdbuf());
// Debugging output.
} else {
cerr << string(WIDTH_DIVIDER, '=') << endl;
cerr << "Starting test: " << test->getName() << endl;
cerr << string(WIDTH_DIVIDER, '-') << endl;
}
TestSuite: Custom output formatting. A large number of improvements have been made by subclassing the CppUnit classes and overriding the base functions, including CompilerOutputter, TestListener, and TextTestRunner. All IO has also been captured by the test runner. The result is that during the test suite run, only the characters '.', 'F', and 'E' are output for a pass, failure and error state. At the end of each test suite category, all failures and errors are reported in full detail, including the different captured IO streams. A final synopsis is printed out, improving the overview in the case of too many tests failing. For the fgCompilerOutputter class, the printSuccess(), printFailureReport(), printFailureDetail(), and printSuiteStats() functions have been replaced to implement the above printout design. The class also stores the std::vector of TestIOCapt structures for the final printouts. The fgTestListener class handles the events from the running of the test suite. The startTest() and endTest() functions are used for IO capture. The IO is placed into a TestIOCapt data structure, with one std::string for holding the combined STDOUT and STDERR IO, and another for the SG_LOG IO. If failures occur, the TestIOCapt structure is appended to the fgCompilerOutputter vector. The startTest() and endTest() functions are also used for starting and stopping a timer to allow the full test suite to be timed. And the addFailure() function simply registers test failures or errors. The fgTestRunner class overrides the CppUnit::TextTestRunner::run() function, simply to prevent the base method from spawning a second test listener, causing the test output to be duplicated. Some auxiliary formatting functions have been added to print out titles, sections, and synopsis summary lines.
2016-02-09 16:34:56 +00:00
// Reset the test status.
m_failure = false;
m_error = false;
m_time = clock();
}
Copy permalink