matlab.unittest.plugins.TestReportPlugin Class

Namespace: matlab.unittest.plugins
Superclasses: matlab.unittest.plugins.TestRunnerPlugin, matlab.unittest.plugins.Parallelizable

Plugin that generates a test report

expand all in page

Description

The matlab.unittest.plugins.TestReportPlugin class provides a plugin that generates a test report. To produce a test report, add an instance of this plugin class to the test runner. A generated test report provides detailed information about tests that passed, failed, remained incomplete, or did not run. For more information about test statuses, see Summary of Test Statuses.

The matlab.unittest.plugins.TestReportPlugin class is a handle class.

Class Attributes

Abstract
true

For information on class attributes, see Class Attributes.

Creation

Create a TestReportPlugin instance using one of its static methods:

  • To create a plugin that generates a DOCX test report, use the producingDOCX static method.

  • To create a plugin that generates an HTML test report, use the producingHTML static method.

  • To create a plugin that generates a PDF test report, use the producingPDF static method.

Properties

expand all

Title of the test report, returned as a string scalar. You can specify the value of this property during creation of the plugin. By default, the plugin uses "MATLAB® Test Report" as the title.

Attributes:

GetAccess
public
SetAccess
immutable

Whether to include the text output from the Command Window, returned as a 0 or 1 of data type logical. You can specify the value of this property during creation of the plugin. By default, the plugin does not include the text output from the Command Window in the test report.

Attributes:

GetAccess
public
SetAccess
immutable

Whether to include the diagnostics for passing events, returned as a 0 or 1 of data type logical. You can specify the value of this property during creation of the plugin. By default, the plugin does not include the diagnostics for passing events in the test report.

Attributes:

GetAccess
public
SetAccess
immutable

Maximum verbosity level of logged diagnostics to include in the test report, returned as a matlab.automation.Verbosity enumeration object. You can specify the value of this property during creation of the plugin. By default, the plugin includes diagnostics logged at the matlab.automation.Verbosity.Terse level.

Logged diagnostics are diagnostics that you supply to the testing framework with the log (TestCase) and log (Fixture) methods.

Attributes:

GetAccess
public
SetAccess
private

Methods

expand all

Examples

collapse all

Create a test suite from two test files, run the suite, and generate a .docx report of the results.

Create a new file in your working folder named ScriptBasedTest.m containing the following test script. The script includes two failing and incomplete tests.

%% Test double class
expSolution = 'double';
actSolution = ones;
assert(isa(actSolution,expSolution))

%% Test single class
expSolution = 'single';
actSolution = ones('single');
assert(isa(actSolution,expSolution))

%% Test uint16 class
expSolution = 'uint16';
actSolution = ones('uint16');
assert(isa(actSolution,expSolution))

%% Test that fails
assert(false==true);

%% Another test that fails
assert(strcmp('correlation','causation'))

Create a file named ClassBasedTest.m containing the following test class.

classdef ClassBasedTest < matlab.unittest.TestCase
    properties (ClassSetupParameter)
        generator = {'twister','combRecursive','multFibonacci'};
    end
    properties (MethodSetupParameter)
        seed = {0,123,4294967295};
    end
    properties (TestParameter)
        dim1 = struct('small',1,'medium',2,'large',3);
        dim2 = struct('small',2,'medium',3,'large',4);
        dim3 = struct('small',3,'medium',4,'large',5);
        type = {'single','double'};
    end
    methods (TestClassSetup)
        function ClassSetup(testCase,generator)
            orig = rng;
            testCase.addTeardown(@rng,orig)
            rng(0, generator)
        end
    end
    methods (TestMethodSetup)
        function MethodSetup(testCase,seed)
            orig = rng;
            testCase.addTeardown(@rng,orig)
            rng(seed)
        end
    end
    methods (Test, ParameterCombination='sequential')
        function testSize(testCase,dim1,dim2,dim3)
            testCase.verifySize(rand(dim1,dim2,dim3),[dim1 dim2 dim3])
        end 
    end
    methods (Test, ParameterCombination='pairwise')
        function testRepeatable(testCase,dim1,dim2,dim3)
            state = rng;
            firstRun = rand(dim1,dim2,dim3);
            rng(state)
            secondRun = rand(dim1,dim2,dim3);
            testCase.verifyEqual(firstRun,secondRun);
        end
    end
    methods (Test)
        function testClass(testCase,dim1,dim2,type)
            testCase.verifyClass(rand(dim1,dim2,type),type)
        end
    end
end

At the command prompt, create a test suite from both test files.

import matlab.unittest.TestRunner;
import matlab.unittest.TestSuite;
import matlab.unittest.plugins.TestReportPlugin;

suite = testsuite({'ScriptBasedTest','ClassBasedTest'})
suite = 

  1×284 Test array with properties:

    Name
    ProcedureName
    TestClass
    BaseFolder
    Parameterization
    SharedTestFixtures
    Tags

Tests Include:
   17 Unique Parameterizations, 0 Shared Test Fixture Classes, 0 Tags.

Create a silent test runner, so that there is no information output to the command window. Create a TestReportPlugin that sends output to the file MyTestReport.docx.

runner = TestRunner.withNoPlugins;
docxFile = 'MyTestReport.docx';
plugin = TestReportPlugin.producingDOCX(docxFile);

Add the plugin to the TestRunner and run the suite.

runner.addPlugin(plugin);
result = runner.run(suite)
Generating report. Please wait.
    Preparing content for the report.
    Adding content to the report.
    Writing report to file.
Report has been saved to: C:\work\MyTestReport.docx

result = 

  1×284 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   282 Passed, 2 Failed, 2 Incomplete.
   0.73477 seconds testing time.

Open the test report.

open(docxFile)

Create a test suite from a function-based test, run the suite, and generate a report of the results. Include passing diagnostics and the text output to the Command Window.

Create a new file in your working folder named FunctionBasedTest.m containing the following function-based test. The test file includes two failing tests.

%% Main function to generate tests
function tests = FunctionBasedTest
tests = functiontests(localfunctions);
end

%% Test Functions
function passingTest(testCase)
actSolution = 13*3+7*5;
expSolution = 74;
verifyEqual(testCase,actSolution,expSolution)
end

function failingTest(testCase)
actSolution = single(1);
verifyTrue(testCase,actSolution)
end

function anotherPassingTest(testCase)
verifyClass(testCase,string('some text'),'string')
end

function anotherFailingTest(testCase)
verifyTrue(testCase,strcmp('42','everything'))
end

At the command prompt, create a test suite from FunctionBasedTest.m. Create a test runner that displays output to the command window using the default plugin.

import matlab.unittest.TestRunner;
import matlab.unittest.TestSuite;
import matlab.unittest.plugins.TestReportPlugin;

suite = testsuite('FunctionBasedTest');
runner = TestRunner.withTextOutput;

Create a TestReportPlugin that sends output to the file MyTestReport2.pdf. Include passing diagnostics and text output from the Command Window in the report.

pdfFile = 'MyTestReport2.pdf';
plugin = TestReportPlugin.producingPDF(pdfFile,...
    'IncludingPassingDiagnostics',true,'IncludingCommandWindowText',true);

Add the plugin to the TestRunner and run the suite.

runner.addPlugin(plugin);
result = runner.run(suite);
Running FunctionBasedTest
.
================================================================================
Verification failed in FunctionBasedTest/failingTest.

    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyTrue failed.
    --> The value must be logical. It is of type "single".
    
    Actual single:
             1

    ------------------
    Stack Information:
    ------------------
    In C:\Work\FunctionBasedTest.m (failingTest) at 15
================================================================================
..
================================================================================
Verification failed in FunctionBasedTest/anotherFailingTest.

    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyTrue failed.
    --> The value must evaluate to "true".
    
    Actual logical:
           0

    ------------------
    Stack Information:
    ------------------
    In C:\Work\FunctionBasedTest.m (anotherFailingTest) at 23
================================================================================
.
Done FunctionBasedTest
__________

Failure Summary:

     Name                                  Failed  Incomplete  Reason(s)
    ===================================================================================
     FunctionBasedTest/failingTest           X                 Failed by verification.
    -----------------------------------------------------------------------------------
     FunctionBasedTest/anotherFailingTest    X                 Failed by verification.
    
Generating report. Please wait.
    Preparing content for the report.
    Adding content to the report.
    Writing report to file.
Report has been saved to: C:\Work\MyTestReport2.pdf

Open the test report.

open(pdfFile)

In a file named FigurePropertiesTest.m in your current folder, create the FigurePropertiesTest test class. The failingTest method of the test class, which intentionally fails, uses a FigureDiagnostic object to save the figure so you can examine it later.

classdef FigurePropertiesTest < matlab.unittest.TestCase
    properties
        TestFigure
    end

    methods (TestMethodSetup)
        function createFigure(testCase)
            testCase.TestFigure = figure;
            testCase.addTeardown(@close,testCase.TestFigure)
        end
    end

    methods (Test)
        function defaultCurrentPoint(testCase)
            cp = testCase.TestFigure.CurrentPoint;
            testCase.verifyEqual(cp,[0 0], ...
                "Default current point must be [0 0].")
        end

        function defaultCurrentObject(testCase)
            import matlab.unittest.constraints.IsEmpty
            co = testCase.TestFigure.CurrentObject;
            testCase.verifyThat(co,IsEmpty, ...
                "Default current object must be empty.")
        end

        function failingTest(testCase)
            import matlab.unittest.diagnostics.FigureDiagnostic
            fig = testCase.TestFigure;
            ax = axes(fig);
            surf(ax,peaks)
            testCase.verifyEmpty(testCase.TestFigure.Children, ...
                FigureDiagnostic(testCase.TestFigure))
        end
    end
end

Import the classes used in this example.

import matlab.unittest.plugins.DiagnosticsRecordingPlugin
import matlab.unittest.plugins.TestReportPlugin

Create a test suite from the test class.

suite = testsuite("FigurePropertiesTest");

Create a test runner that records diagnostics and generates a PDF test report.

runner = testrunner("minimal");
runner.addPlugin(DiagnosticsRecordingPlugin)
runner.addPlugin(TestReportPlugin.producingPDF("report.pdf"))

Set the ArtifactsRootFolder property of the test runner so that the framework saves any diagnostic artifacts produced during the test run to your current folder.

runner.ArtifactsRootFolder = pwd;

Run the tests in the test suite. One of the tests fails.

results = runner.run(suite)
Generating test report. Please wait.
    Preparing content for the test report.
    Adding content to the test report.
    Writing test report to file.
Test report has been saved to:
 C:\work\report.pdf

results = 

  1×3 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   2 Passed, 1 Failed (rerun), 0 Incomplete.
   1.6241 seconds testing time.

Display the diagnostic result for the failed test. The returned DiagnosticResult object indicates that the framework saved two artifacts related to the third test. By default, a FigureDiagnostic object results in both a FIG file and a PNG file.

results(3).Details.DiagnosticRecord.TestDiagnosticResults
ans = 

  DiagnosticResult with properties:

         Artifacts: [1×2 matlab.automation.diagnostics.FileArtifact]
    DiagnosticText: 'Figure saved to:↵--> C:\Users\username\AppData\Local\Temp\817b4071-9d9d-471e-bead-eb0f93a89f1b\Figure_c79a5df0-04fc-4ebc-9eb6-920b23feba40.fig↵--> C:\Users\username\AppData\Local\Temp\817b4071-9d9d-471e-bead-eb0f93a89f1b\Figure_c79a5df0-04fc-4ebc-9eb6-920b23feba40.png'

Display the location of the first artifact. To inspect the image related to the failed test, open the file at the location shown in the FullPath property. The generated PDF test report also includes the captured image and the artifact paths.

results(3).Details.DiagnosticRecord.TestDiagnosticResults.Artifacts(1)
ans = 

  FileArtifact with properties:

        Name: "Figure_c79a5df0-04fc-4ebc-9eb6-920b23feba40.fig"
    Location: "C:\Users\username\AppData\Local\Temp\817b4071-9d9d-471e-bead-eb0f93a89f1b"
    FullPath: "C:\Users\username\AppData\Local\Temp\817b4071-9d9d-471e-bead-eb0f93a89f1b\Figure_c79a5df0-04fc-4ebc-9eb6-920b23feba40.fig"

More About

expand all

Version History

Introduced in R2016b

expand all

See Also

Classes

Functions