Simulink Design Verifier Test Generation Internal Error

Hallo,

I am trying to using the Simulink Design Verifier Toolbox on a large model, but I am facing some challenges and errors which I am not able to solve. I have started trying the test generation of SLDV, which from the beginning seemed to be extremely slow and not feasible in a reasonable amount of time. I have anyway tried to apply some changes to generate the test a bit faster (like fixing input range), but no extreme improvement could be observed. Afterwards I then started trying to just let it run as much as I can to see if it ultimately can generate what I need and unfortunately now it is giving me always the same error after a couple of minutes (6-20 minutes).

The error is the following:

Looking at previously answered questions (https://ch.mathworks.com/matlabcentral/answers/320807-design-verifier-error-after-2-days-of-running-analysis), I've seen that the problem may comes from a lack of RAM memory. I have tried the to check the memory consumption, but it looks like it never exceeds 50% of the total memory.

I would also ask some advices on how to use simulink design verifier with large models, because from what I've seen up to now it doesn't looks really promising....

Regards,

Davide

Answer 1

[Pat Canny]

Hi Davide,

Thanks for following up on the error.

Regarding your question about large model support, what do you mean by "doesn't look very promising"? Are you seeing very long analysis times?

Which workflow are you using with Design Verifier? (Design Error Detection, Test Generation, or Property Proving)

If you are performing automatic test generation, I suggest trying the Test Generation Advisor: https://www.mathworks.com/help/sldv/ug/select-components-for-test-case-generation.html

Here are a few other tips, though I would be happy to chat about this:

1. If your model contains a lot of timer elements (e.g. delay blocks or after statements in Stateflow), this can add significant analysis time. Consider shortening the length of the timers in the model.
2. If your model contains a lot of floating point operations, this can also add significant analysis time. Consider casting to single-precision (if your signals are doubles) or, if possible, to int.
3. Wherever possible, split the logical parts of your design from the parts with a lot of math operations for analysis. Make use of harnesses or Model Slicer. Parts of your model with a lot of Stateflow tend to be easier to analyze, as well.

[Davide Frey]

Exactly: mostly really long analysis time (or even impossibility to get the analysis concluded, due to PC crashing after a while). I can encounter the really long analysis time for both Design Error Detection (specifically for Inf/Nan and Subnormal floating-point tests) and for Test Generation.

I am trying to perform automatic test generation, in order to have a set of unit-tests for a model that can then be runned later, when changes are made to the model. Ideally the best thing for me would be to be able to:

• Have a "baseline" test set that achieve 100% coverage of the model (using automatic test generation).
• Be able to manually add other tests that can test specific model dynamics or model requirements.

Regarding this second element (the manual addition of tests): as of now, I am trying to do it by adding the test to the test harness signal builder, to later store in the .mat file (sldvdata) so that I can call it using "sldvruntest" and then comparing the outputs. Is it possible to do something like this with some simulink tool?

I thank you very much for the tips, here's some comments/questions on them:

1. It contains a lot of delays, what you mean by "shortening the length" of them?
2. It does that as well, though I don't think I can switch to single or int.
3. By splitting you mean really separate the model in sort of a "logic part" and a "math part", test them separately and then use them together with model referencing? In the model we have a lot of logical components, would it be better to switch to Stateflow then?

Best, Davide

[Pat Canny]

Hi Davide,

These are great questions. I can help you out on a few of these items. I have passed along your specific Test Generation workflow questions to MathWorks Development and our Pilot Engineering team, who will answer those directly here. This is a relatively common workflow, but I want to be sure I get the right folks to answer.

To answer your last three questions:

"It contains a lot of delays, what you mean by "shortening the length" of them?"

By "shortening", I mean reduce the number of simulation steps before, for example, a transition is taken. As an example, if you specify a number of ticks in a Stateflow chart using an "after" statement, then the number of ticks is what you would reduce.

"By splitting you mean really separate the model in sort of a "logic part" and a "math part", test them separately and then use them together with model referencing? In the model we have a lot of logical components, would it be better to switch to Stateflow then?"

Yes, you could use model reference to recombine the components. That is a good practice.

Design Verifier tends to work well with Stateflow, especially for designs with a lot of logic (which is what Stateflow does best). However, if your design uses a lot of standard Simulink logic components, it wouldn't necessarily save you on analysis time. If you're interested, we could chat directly about your design and workflow. Feel free to reach out to me: [e-mail]

[Davide Frey]

Ok thank you, I will reach you out to discuss further then.

Thank you for your time.

Answer 2

[Davide Frey]

It seems that this error was depending on the parameters defined in the file sldv_param.m

If multiple parameter packages are defined, those package needs to be consistent with each other (i.e needs to have the same elements). Otherwise you could get this error.

Answer 3

[Prahladavaradan Sampath]

hi Davide,

I will try to add on to the suggestions Pat has already provided, but first I would like to understand a bit more about the internal error. Could you provide the output of the command-line invocation of SLDV:

[s,f,h,msg] = sldvrun('modelname');
save msg.mat msg

The structure "msg" would have some more hints about the error that we could use to help you.

regards

Prahlad

[Davide Frey]

Hi Prahlad,

Unfortunately at the moment I don't have time to look back at when the error was there or to reproduce it. As soon as I have a bit of time to focus on that, I will try to see what kind of msg I was getting and I will let you know.

Best,

Davide

[UPDATE1]: I've tried to replicate this error but unfortuntely I can not manage to get it again. This time the analysis quits before triggering the internal error but tells me:

"Analysis ran out of memory"

If I will manage to replicate it, I'll let you know.

[UPDATE2]: Also these latest error seems that was caused by the mis-definition of parameters, especially the sample time of the model was defined mistakenly as a parameter and simulink was changing it's value and maybe also size during the test generation, causing this failure and a not readable coverage data!

[UPDATE3]: Nevermind, I tried again to run a full analysis and after 1:34:53 it stopped again saying "Analysis ran out of memory". I've managed to create a msg.mat file (attached), incase it helps. Is there any workaround to this error?

Answer 4

[Prahladavaradan Sampath]

hi Davide,

You could try using sldvmergeharness to combine the sldv generated harness with your custom test harness models. The overall syntax is

sldvmergeharness('target_harness',{'sldv_harness','harness1'},{'','param1 = 4; param2 = 5'});

The third argument could be used to initialize parameters for the tests in the corresponding harness.

You should be able to do this incrementally. For example after the above step, you could do:

sldvmergeharness('target_harness',{'target_harness','harness2'},{'','param1=5; param2=7'});

This would accumulate the tests in harness2 into target_harness.

regards

Prahlad

[Davide Frey]

Hi Prahlad,

I was already trying to use sldvmergeharness, and for the test harness itself it seems to be working just fine. As of now, I am doing the following:

• Generate the tests using sldvrun() with opts.Parameters set to "on"
• Create new tests with custom parameters and custom signals, and store them in the sldvdata.mat file, generated from sldvrun().

This allows me also to evaluate the new total coverage, using sldvruntest(). I would like to forward you to one of my latest question, were I written down more in detail what my problem with the custom test generation is:

https://ch.mathworks.com/matlabcentral/answers/439175-from-where-is-a-test-harness-taking-information-on-parameters?s_tid=prof_contriblnk

Essentially I don't know how to add in the new test harness the information about the parameters.

I hope that I explained my problems better.

Thank you for your help and best regards,

Davide

[Pat Canny]

Hi Davide,

We'd like to learn more about your workflow and help get you unstuck.

When you have time, please e-mail me at [e-mail]

Thanks.