Main Content

Resolve Missing Artifacts, Links, and Results in the Model Testing Dashboard

Issue

The Model Testing Dashboard analyzes artifacts—models, requirements, tests, and results—that are part of the requirements-based testing workflow for software unit models. If an artifact or a link between artifacts is not part of the requirements-based testing workflow, it might not appear in the Model Testing Dashboard or contribute to the analysis results. Additionally, some artifacts and links are not supported by the Model Testing Dashboard. If you expect a link or artifact to appear in the dashboard and it does not, try one of these solutions.

Possible Solutions

Try these solutions when you begin troubleshooting artifacts in the dashboard:

  • Save changes to your artifact files.

  • Check that your artifacts are saved in the project. The dashboard does not analyze files that are not saved in the project.

  • Check that your artifacts are not in a referenced project. The dashboard does not analyze files in referenced projects.

  • Check that your artifacts are on the MATLAB® search path before you open the dashboard. When you change the MATLAB search path, the traceability information in the Artifacts panel is not updated. Do not change the search path while the dashboard is open.

  • Open the Diagnostics pane and address errors or warnings.

  • Use the dashboard to re-trace the artifacts and re-collect metric results.

Depending on the type of artifact or analysis issue that you are troubleshooting, try one of these solutions.

Enable Artifact Tracing for the Project

As you edit and save the artifacts in your project, the dashboard needs to track these changes to enable artifact tracing and to detect stale results.

By default, the dashboard requests that you enable artifact tracing the first time you open a project in the dashboard. Click Enable and Continue to allow the dashboard to track tool outputs to detect outdated metric results.

The dashboard needs to track tool outputs, such as test results from Simulink® Test™, to detect outdated metric results.

You can also enable artifact tracing from the Startup and Shutdown settings of the project. In the Startup and Shutdown settings for your project, select Track tool outputs to detect outdated results. For more information on the tool outputs and outdated metric results, see Digital Thread.

Cache Folder Artifact Tracking

By default, projects use the same root folder for both the simulation cache folder and the code generation folder. If possible, use different root folders for the simulation cache folder and code generation folder in your project. When you specify different root folders, the dashboard no longer needs to track changes to the simulation cache folder.

To view the cache folder settings for your project, on the Project tab, in the Environment section, click Details. The Project Details dialog shows the root folders specified for the Simulation cache folder and Code generation folder.

The behavior of change tracking only depends on the project settings. Custom manipulations do not impact the change tracking behavior. For example, the dashboard does not check root folders specified by Simulink.fileGenControl.

Project Requires Analysis by the Dashboard

The first time that you open the dashboard for the project, the dashboard identifies the artifacts in the project and collects traceability information. The dashboard must perform this first-time setup to establish the traceability data before it can monitor the artifacts. If you cancel the first-time setup, the artifacts in the project appear in the Unanalyzed folder in the Artifacts panel. To trace the unanalyzed artifacts, click Collect > Trace Artifacts.

Incorrect List of Models in Project Panel

The Project panel shows the models in your project that are either unit models or component models. Models are organized under the components that reference them, according to the model reference hierarchy. If the list of unit and component models does not show the expected hierarchy of your models, try one of these solutions.

Check that your unit and component models are labeled.  Label the unit and component models in your project and configure the dashboard to recognize the labeled models. Note that if a unit model references one or more other models, the referenced models appear in the Design folder under the unit model. For more information about labeling models and configuring the dashboard, see Categorize Models in a Hierarchy as Components or Units. Check that if you have Observer models, they are not labeled as units. The dashboard includes Observer models as units if they match the label requirements.

Check that your model was saved in a supported release.  Check that your model was saved in R2012b or later. The dashboard does not support models that were saved before R2012b.

Block Skipped During Artifact Analysis

If a block has a mask and the mask hides the content of the block, the dashboard excludes the block from artifact analysis.

Check that your custom libraries do not contain blocks with self-modifiable masks. The dashboard does not analyze blocks that contain self-modifiable masks. Self-modifiable masks can change the structural content of a block, which is incompatible with artifact traceability analysis.

Library Missing from Artifacts Panel

Check that the library does not use a library forwarding table. The dashboard does not support library forwarding tables.

Requirement Missing from Artifacts Panel

If a requirement is missing from the Artifacts panel, try one of these solutions.

Check that the requirement is a functional requirement.  Verify that the requirement is configured as a functional requirement. In the Requirement Editor, on the left pane, click the requirement. On the right pane, in the Properties section, set Type to Functional. Because the Model Testing Dashboard reports on requirements-based unit testing, only functional requirements appear in the Artifacts panel and are analyzed by the dashboard.

Check that the requirement is saved in a supported requirements file.  Verify that the requirement is saved in a requirements file that has the .slreqx extension.

Test Case Missing from Artifacts Panel

Check that the test case is supported by the Model Testing Dashboard. The Model Testing Dashboard does not support MATLAB-based Simulink tests.

Test Harness Missing from Artifacts Panel

Check that the test harness is not an internal test harness for a System Composer™ architecture model. The dashboard does not support internal test harnesses for System Composer architecture models. If your model already uses internal test harnesses, you can convert the internal test harnesses to externally stored test harnesses. Navigate to the top of the main model and open Simulink Test. On the Tests tab, click Manage Test Harnesses > Convert to External Harnesses. Click OK to convert the affected test harnesses. External test harnesses for System Composer architecture models appear in the Artifacts panel in the subfolder Tests > Test Harnesses.

Check that the test harness is not on a subsystem inside a library block instance. If a test harness is on a subsystem inside a library block inside a model, the dashboard cannot perform artifact traceability analysis on the test harness. The relationship between a model and a test harness on a subsystem inside a library block instance is incompatible with artifact traceability analysis. To enable artifact traceability analysis, move the test harness to the library.

Test Result Missing from Artifacts Panel

Check that either:

  • The result is saved in a test results file. Save test results by exporting them from the Test Manager.

  • You collected the results during the current project session and have not closed them. When you collect test results and do not export them, the dashboard recognizes the temporary results in the Test Manager, denoted by the Temporary test results icon icon. The dashboard stops recognizing the temporary results when you close the project, close the test results set, or export the test results to a results file.

Artifact Returns a Warning

Check the details of the warning by opening the Diagnostics pane.

Artifact Returns an Error

Check the details of the error by opening the Diagnostics pane.

If the dashboard returns an error in the Diagnostics pane, the metric data shown by the dashboard widgets may be incomplete. Errors indicates that the dashboard may not have been able to properly trace artifacts, analyze artifacts, or collect metrics.

Before using the metrics results shown in the dashboard, resolve the reported errors and retrace the artifacts.

Fix ambiguous links.  Check that the links in your project define unambiguous relationships between project artifacts.

In requirements-based testing, projects often contain links between software requirements and:

  • the design artifacts that implement the requirements

  • the test cases that test the implemented requirements

  • the higher-level system requirements

The links in your project help to define the relationships between artifacts. The dashboard uses a digital thread to capture the traceability relationships between the artifacts in your project. To maintain the traceability relationships, the dashboard returns an error when the links to project artifacts are ambiguous. Ambiguous links are not supported in the dashboard.

If one of these conditions is met, the dashboard cannot establish unambiguous traceability:

  • A link set shadows another loaded link set of the same name.

  • A requirement set shadows another loaded requirement set of the same name.

  • A link is not on the project path or is only temporarily on the project path.

  • A link is not portable.

    To avoid links that are not portable:

    • Do not set the preference for a link path format to be an absolute path. Absolute paths are not portable. For information on how to set the preference for the path format of links, see rmipref (Requirements Toolbox) and Document Path Storage (Requirements Toolbox).

    • When you identify the source artifact of a link set, use the default link file name and location. Link source remapping persists in the MATLAB preferences directory and is not portable. For more information, see Requirements Link Storage (Requirements Toolbox).

Use the details and suggested actions in the dashboard error messages to fix the ambiguous links.

For more information on traceability relationships and the digital thread, see Digital Thread.

Trace Issues

If an artifact appears in the Trace Issues folder when you expect it to trace to a unit model, depending on the type of artifact that is untraced, try one of these solutions.

Fix an untraced requirement.  Check that the requirement traces to the unit model using an implementation link. The requirement and its links must meet one of these criteria:

  • The requirement is linked to the model or to a library subsystem used by the model with a link where the Type is set to Implements.

  • The requirement is the child of a container requirement that is linked to the model or to a library subsystem used by the model with a link where the Type is set to Implements.

  • The requirement traces to the model through a combination of the previous two criteria. For example, a requirement that is under a container requirement that links to another requirement, which links to the model.

Requirements-based testing verifies that your model fulfills the functional requirements that it implements. Because the Model Testing Dashboard reports on requirements-based testing quality, it analyzes only requirements that are specified as functional requirements and implemented in the unit. For each unit, the dashboard shows the functional requirements that are implemented in the unit in the folder Functional Requirements > Implemented.

Check that the requirement does not use an unsupported link. The Model Testing Dashboard does not trace these links:

  • Downstream links. The Model Testing Dashboard traces only links from lower-level requirements to higher-level requirements. For each unit, the dashboard shows the higher-level functional requirements in the folder Functional Requirements > Upstream. Upstream links are directly or transitively linked to the implemented functional requirements for the unit. Check the direction of the link by using the Requirements Editor. In the Requirements Editor, in the Analysis section, click Traceability Diagram. In the Impact Analysis section, use the buttons Upstream and Downstream to check the direction of the links.

  • Embedded links, which are requirements files that are saved directly in the model file.

  • Links to requirements that are saved externally and linked using the Requirements Management Interface (RMI).

  • Links to MATLAB code files.

  • Links to MATLAB Function blocks if you do not have a Stateflow® license. Analyzing MATLAB Function blocks requires a Stateflow license.

  • Links to some Stateflow elements.

  • Links in deprecated requirement files, which have the extension .req. To analyze requirement links in the dashboard, save the links in an .slmx file or create them in the requirements file (.slreqx) that has the requirements.

  • Links to models for which the model file extension changed. If a requirement is linked to a model with the file extension .slx, but the model file extension is changed to .mdl, the dashboard lists the requirement link as unresolved. Modify the requirement link to reference the expected model file and re-save the requirement link.

  • Symbolic file links in a project, such as shortcuts.

  • Links to modeling elements that are not supported by the Model Testing Dashboard, such as library forwarding tables.

For requirement links to data dictionary entries, the dashboard traces from the requirement to the data dictionary file associated with the data dictionary entry.

Fix an untraced requirement link set.  Check that the requirement link set does not use the legacy Requirements Management Interface (RMI) format. To allow the dashboard to analyze your requirement link set, pass your requirement link set as the input argument to the function slreq.refreshSourceArtifactPath.

Fix an untraced design artifact.  Check that the design artifact does not rely on a model callback to be linked with the model. The Model Testing Dashboard does not execute model loading callbacks when it loads the models for analysis. If a model relies on a callback to link a data dictionary, the data dictionary will not be linked when the dashboard runs the traceability analysis.

Fix an untraced test case.  Check that the test case runs on the model or runs on a subsystem in the model by using a test harness.

Fix an untraced test result.  Check that the project and test case are set up correctly and re-run your tests. If one of these conditions is met when you run your test case, the generated results are untraced because the dashboard cannot establish unambiguous traceability to the unit:

  • No project is loaded.

  • Artifact tracing is not enabled for the project. If artifact tracing is not enabled, the dashboard cannot track changes or trace from the test cases to the generated test results. For more information, see Enable Artifact Tracing for the Project.

  • You do not have a Simulink Check™ license.

  • The test file is stored outside the project.

  • The test file has unsaved changes.

  • The tested model has unsaved changes.

  • The test file returns an error during traceability analysis.

  • The tested model returns an error during traceability analysis.

  • The test result comes from a test case that is not supported by the Model Testing Dashboard, such as a MATLAB-based Simulink test.

Check that the results and environment are set up correctly and re-export your test results. If one of these conditions is met when you export your test results, the generated results are untraced because the dashboard cannot establish unambiguous traceability to the unit:

  • No project is loaded.

  • Artifact tracing is not enabled for the project. For more information, see Enable Artifact Tracing for the Project.

  • You do not have a Simulink Check license.

  • The test result file returns an error during traceability analysis.

Metric Does Not Report Results for Requirement, Test Case, or Test Result

If an artifact traces to one of your unit models but does not appear in the metric results for that unit, depending on the type of artifact, try one of these solutions.

Fix a requirement that does not produce metric results.  Check that the requirement directly links to the model with a link where the Type is set to Implements. The dashboard metrics analyze only implemented functional requirements. For each unit, the implemented functional requirements appear in the folder Functional Requirements > Implemented. Upstream requirements appear in the folder Functional Requirements > Upstream, but do not contribute to the metric results because upstream requirements are only indirectly or transitively linked to the implemented requirements.

Fix a test case that does not produce metric results.  Check that the test case directly tests either the entire unit model or the model subsystems. The dashboard metrics analyze only unit tests. For each unit, the unit test cases appear in the folder Tests > Unit Tests. Other test cases, that are not unit tests, appear in the folder Tests > Others, but do not contribute to the metric results because other tests do not directly test the unit model or the model subsystems. For example, the dashboard does not consider tests on a library to be unit tests.

Fix a test result that does not produce metric results.  Check that the results meet these criteria:

  • The results are the most recent results generated from the test cases.

  • The results are from unit tests which appear in the folder Tests > Unit Tests. Test cases in the folder Tests > Others do not contribute to the metric results.

If a test case includes multiple iterations, the metric results reflect the status of the whole test case and do not show individual iteration results.

The dashboard can only isolate outdated results to individual test cases or test suites if the test cases or test suites have revision numbers. If a test case or test suite was saved in a release that did not save revision numbers, use the function sltest.testmanager.refreshTestRevisions on the test file to refresh the revision information.

The coverage metrics do not aggregate coverage for external MATLAB files and the coverage metrics do not include coverage results from external MATLAB code called from the component.

For each unit, the test results that produce metric results appear in the folder Test Results > Unit Simulation. The test results in the folder Test Results > Others do not contribute to the metric results.

Fix a test that does not produce simulation test result analysis metric results.  Check that the test case is a unit test and produces simulation results. For each unit, the metrics analyze the test results in the folder Test Results > Unit Simulation. The test results in the folder Test Results > Others are results that are not simulation results, are not from unit tests, or are only reports. For example, SIL results are not simulation results. The metrics in the Simulation Test Result Analysis section count results from only simulation tests, whereas the metrics in the Test Case Analysis section count all unit tests.

If a test is not counted in the metrics in the Simulation Test Result Analysis section, check that the test case meets these criteria for being a simulation test:

  • The simulation mode is Normal, Accelerator, or Rapid Accelerator. If the test uses iterations to set a different simulation mode after testing one of these modes, the test is still considered a simulation test.

  • The test is not a real-time test.

  • If the test is an equivalence test, the first simulation meets one of the first two criteria.

  • If the test contains multiple iterations, the test case or at least one iteration meets one of the first two criteria.

Metric Result Shows a Missing Link or Artifact

The metric results do not count all types of traceability links. If a metric shows that a test case or requirement is missing links when you expect it to be linked, try one of these solutions.

Fix a link that is not counted in traceability results.  If there is a link between a requirement and a test case, but the traceability metrics show that the test or requirement is unlinked, check if the link is supported by the dashboard metrics. The metrics do not support these links:

  • A requirement link to a justification. If a requirement is linked with a justification and not linked to a test case, it appears as unlinked in the metric results.

  • A requirement link to a test harness.

Fix missing model coverage in test results.  If the model coverage metrics report coverage gaps that you do not expect, re-run the test cases and re-collect the metric results for the new test results. The Model Testing Dashboard might show coverage gaps if:

  • You change the test results file or the coverage filter file after you collect the metrics, including if you re-import the test results file after you make changes.

  • You collect accumulated coverage results and make changes to the model file after running one or more tests.

See Also

Related Topics