Using sim Function Within parfor

Note

Running parallel simulations by calling the sim function inside a parfor loop is not recommended. To run parallel simulations, use the parsim function. For more information, see Run Parallel Simulations.

The parfor command allows you to run simultaneous, parallel simulations of your models. In this context, parallel means that multiple simulations of the same model run at the same time, with each concurrent simulation running on a different worker. Running simulations in parallel often helps for performing multiple simulations of the same model for different inputs or for different parameter settings. For example, you can save simulation time when performing parameter sweeps and Monte Carlo analyses by running the simulations in parallel. Running parallel simulations using parfor does not support decomposing your model into smaller connected pieces and running the individual pieces of the same simulation on different workers.

Normal, accelerator, and rapid accelerator simulation modes are supported by sim in parfor. For other simulation modes, you need to address any workspace access issues and data concurrency issues to produce useful results. Specifically, the simulations need to create separately named output files and workspace variables. Otherwise, each simulation overwrites the same workspace variables and files, or can have collisions trying to write variables and files simultaneously.

For more information about running simulations in accelerator and rapid accelerator modes, see:

Also, see parfor (Parallel Computing Toolbox).

Note

If you open models inside a parfor statement, close them again using the command bdclose all to avoid leaving temporary files behind.

Normal Mode Simulation with sim in parfor

This code fragment shows how you can use sim and parfor in normal mode. Save changes to your model before simulating in parfor. The saved copy of your model is distributed to parallel workers when simulating in parfor.

% 1) Load model and initialize the pool.
openExample('sldemo_suspn_3dof');
mdl = 'sldemo_suspn_3dof';
load_system(mdl);
parpool;

% 2) Set up the iterations that we want to compute.
Cf                  = evalin('base','Cf');
Cf_sweep            = Cf*(0.05:0.1:0.95);
iterations          = length(Cf_sweep);
simout(iterations)  = Simulink.SimulationOutput;

% 3) Need to switch all workers to a separate tempdir in case 
% any code is generated for instance for StateFlow, or any other 
% file artifacts are  created by the model.
spmd
    % Set up tempdir and cd into it
    currDir = pwd;
    addpath(currDir);
    tmpDir = tempname;
    mkdir(tmpDir);
    cd(tmpDir);
    % Load the model on the worker
    load_system(mdl)
end

% 4) Loop over the number of iterations and perform the
% computation for different parameter values.
parfor idx=1:iterations   
    set_param([mdl '/Road-Suspension Interaction'],'MaskValues',...
        {'Kf',num2str(Cf_sweep(idx)),'Kr','Cr'});
    simout(idx) = sim(mdl,'SimulationMode','normal');
end

% 5) Switch all of the workers back to their original folder.
spmd
    cd(currDir);
    rmdir(tmpDir,'s');
    rmpath(currDir);
    close_system(mdl,0);
end

close_system(mdl,0);
delete(gcp('nocreate'));

Normal Mode Simulation with sim in parfor and MATLAB Parallel Server

To run normal mode simulations using sim in parfor with MATLAB® Parallel Server™, make these modifications to the code from Normal Mode Simulation with sim in parfor:

  • Modify the parpool command to name the parallel pool and create an object you can use to reference the pool.

    p = parpool('clusterProfile');

  • After you create the parallel pool, attach the files the model requires to run to the pool for distribution to the workers.

    files = dependencies.fileDependencyAnalysis(mdl);
p.addAttachedFiles(files);

  • If you do not have a MATLAB Parallel Server cluster, use your local cluster. For more information, see Discover Clusters and Use Cluster Profiles (Parallel Computing Toolbox).

Start your cluster before running the code.

% 1) Load model and initialize the pool.
openExample('sldemo_suspn_3dof');
mdl = 'sldemo_suspn_3dof';
load_system(mdl);
parpool;

% 2) Set up the iterations that we want to compute.
Cf                  = evalin('base','Cf');
Cf_sweep            = Cf*(0.05:0.1:0.95);
iterations          = length(Cf_sweep);
simout(iterations)  = Simulink.SimulationOutput;

% 3) Need to switch all workers to a separate tempdir in case 
% any code is generated for instance for StateFlow, or any other 
% file artifacts are  created by the model.
spmd
    % Set up tempdir and cd into it
    addpath(pwd);
    currDir = pwd;
    addpath(currDir);
    tmpDir = tempname;
    mkdir(tmpDir);
    cd(tmpDir);
    % Load the model on the worker
    load_system(model);
end

% 4) Loop over the number of iterations and perform the
% computation for different parameter values.
parfor idx=1:iterations   
    set_param([mdl '/Road-Suspension Interaction'],'MaskValues',...
        {'Kf',num2str(Cf_sweep(idx)),'Kr','Cr'});
    simout(idx) = sim(model,'SimulationMode','normal');
end

% 5) Switch all of the workers back to their original folder.
spmd
    cd(currDir);
    rmdir(tmpDir,'s');
    rmpath(currDir);
    close_system(model,0);
end

close_system(mdl,0);
delete(gcp('nocreate'));

Rapid Accelerator Simulation with sim in parfor

Running rapid accelerator simulations inside a parfor loop combines speed with automatic distribution of a prebuilt executable to the parallel workers. As a result, this mode eliminates duplication of the update diagram phase.

To run parallel rapid accelerator simulations using the sim function inside a parfor loop:

  • Set the simulation mode of the model to rapid accelerator..

  • Save changes to your model before simulating in parfor. The saved copy of your model is distributed to parallel workers when simulating in parfor.

  • Ensure that the rapid accelerator target is already built and up to date.

  • Disable the rapid accelerator target up-to-date check by specifying the name-value argument RapidAcceleratorUpToDateCheck as 'off' in the call to sim.

To satisfy the second condition, you can change parameters only between simulations that do not require a model rebuild. In other words, for the parameter value to change, the structural checksum of the model must remain the same. Hence, you can change only tunable block diagram parameters and tunable run-time block parameters between simulations. For a discussion on tunable parameters that do not require a rebuild subsequent to their modifications, see Determine Whether Change Requires Rebuild.

To disable the rapid accelerator target up-to-date check, use the sim function, as shown in this code.

parpool;
% Load the model and set parameters
mdl = 'vdp';
load_system(mdl);
% Build the rapid accelerator target
rtp = Simulink.BlockDiagram.buildRapidAcceleratorTarget(mdl);
% Run parallel simulations
parfor i=1:4
   simOut{i} = sim(mdl,'SimulationMode','rapid',...
               'RapidAcceleratorUpToDateCheck','off',...
               'SaveTime','on',...
               'StopTime',num2str(10*i));
   close_system(mdl,0);
delete(gcp('nocreate'));

In this example, the call to the buildRapidAcceleratorTarget function generates the rapid accelerator target once. Subsequent calls to sim with the RapidAcceleratorUpToDateCheck option off guarantees that the rapid accelerator target is not regenerated, which resolves data concurrency issues.

When you disable the rapid accelerator up-to-date check, changes that you make to block parameter values in the model, for example, by using Block Parameters dialog boxes, by using the set_param function, or by changing the values of MATLAB variables, do not affect the simulation. To pass new parameter values to the simulation, use the RapidAcceleratorParameterSets name-value argument.

Workspace Access Issues

Workspace Access for MATLAB Workers

By default, to call sim in parfor, a parallel pool opens automatically, enabling the simulations to run in parallel. Alternatively, you can also first open MATLAB workers using the parpool function. The parfor function then runs the code within the parfor loop in these MATLAB worker sessions. The MATLAB workers, however, do not have access to the workspace of the MATLAB client session where the model and its associated workspace variables have been loaded. Hence, if you load a model and define its associated workspace variables outside of and before a parfor loop, then neither is the model loaded, nor are the workspace variables defined in the MATLAB worker sessions where the parfor iterations are executed. This is typically the case when you define model parameters or external inputs in the base workspace of the client session. These scenarios constitute workspace access issues.

Transparency Violations

When you call sim in parfor with srcWorkspace set to current, the simulation uses the parfor workspace, which is a transparent workspace. The software issues an error for transparency violation. For more information on transparent workspaces, see Ensure Transparency in parfor-Loops or spmd Statements (Parallel Computing Toolbox).

Data Dictionary Access

When a model is linked to a data dictionary, to write code in parfor that accesses a variable or object that you store in the dictionary, you must use the functions Simulink.data.dictionary.setupWorkerCache and Simulink.data.dictionary.cleanupWorkerCache to prevent access issues. For an example, see Sweep Variant Control Using Parallel Simulation. For more information, see What Is a Data Dictionary?.

Resolving Workspace Access Issues

When a model is loaded into memory in a MATLAB client session, the model is only visible and accessible in that MATLAB session and is not accessible in the memory of the MATLAB sessions on the workers. Similarly, the workspace variables associated with a model that are defined in a MATLAB client session (such as parameters and external inputs) are not automatically available in the worker sessions. You must ensure that the model is loaded and that the workspace variables referenced in the model are defined in the MATLAB session on each worker using these options:

  • In the parfor loop, use the sim function to load the model and to set parameters that change with each iteration. Alternatively, load the model and then use the get_param and set_param functions to get and set the parameters within the parfor loop.

  • In the parfor loop, use the MATLAB functions evalin and assignin to assign values to variables.

Alternatively, you can simplify the management of workspace variables by defining them in the model workspace. These variables will then be available when the model is loaded into the worker sessions. There are limitations associated with using the model workspace. For example, you cannot store Simulink.Signal objects that use a storage class other than Auto in a model workspace. For a detailed discussion on the model workspace, see Model Workspaces.

Specify Parameter Values as Arguments for sim Function

Use the sim function in the parfor loop to set parameters that change with each iteration.

mdl = 'vdp';
load_system(mdl)
    
%Specify parameter values. 
paramName = 'StopTime';
paramValue = {'10' '20' '30' '40'};
    
% Run parallel simulations
parfor i=1:4
    simOut{i} = sim(mdl,...
                    paramName,paramValue{i},...
                    'SaveTime','on');
end
    
close_system(mdl,0);

An equivalent option is to load the model and then use the set_param function to set the paramName in the parfor loop.

Specify Variable Values Using assignin

You can pass the values of model or simulation variables to the MATLAB workers using the assignin or evalin functions. This code shows how to use this technique to load variable values into the appropriate workspace of the MATLAB workers.

parfor i = 1:4
    assignin('base','extInp',paramValue{i});
    simOut{i} = sim(model,'ExternalInput','extInp');
end

Sweep Variant Control Using Parallel Simulation

To use parallel simulation to sweep a variant control (a Simulink.Parameter object whose value influences the variant condition of a Simulink.VariantExpression object) that you store in a data dictionary, use this code as a template. Change the names and values of the model, data dictionary, and variant control to match your application.

To sweep block parameter values or the values of workspace variables that you use to set block parameters, use Simulink.SimulationInput objects instead of the programmatic interface to the data dictionary. See Optimize, Estimate, and Sweep Block Parameter Values.

You must have a Parallel Computing Toolbox™ license to perform parallel simulation.

% For convenience, define names of model and data dictionary
model = 'mySweepMdl';
dd = 'mySweepDD.sldd';

% Define the sweeping values for the variant control
CtrlValues = [1 2 3 4];

% Grant each worker in the parallel pool an independent data dictionary 
% so they can use the data without interference
spmd 
    Simulink.data.dictionary.setupWorkerCache
end

% Determine the number of times to simulate
numberOfSims = length(CtrlValues);

% Prepare a nondistributed array to contain simulation output
simOut = cell(1,numberOfSims);

parfor index = 1:numberOfSims
    % Create objects to interact with dictionary data
    % You must create these objects for every iteration of the parfor-loop
    dictObj = Simulink.data.dictionary.open(dd);
    sectObj = getSection(dictObj,'Design Data');
    entryObj = getEntry(sectObj,'MODE'); 
    % Suppose MODE is a Simulink.Parameter object stored in the data dictionary
    
    % Modify the value of MODE
    temp = getValue(entryObj);
    temp.Value = CtrlValues(index);
    setValue(entryObj,temp);

    % Simulate and store simulation output in the nondistributed array
    simOut{index} = sim(model);
    
    % Each worker must discard all changes to the data dictionary and
    % close the dictionary when finished with an iteration of the parfor-loop
    discardChanges(dictObj);
    close(dictObj);
end

% Restore default settings that were changed by the function
% Simulink.data.dictionary.setupWorkerCache
% Prior to calling cleanupWorkerCache, close the model

spmd
    bdclose(model)
    Simulink.data.dictionary.cleanupWorkerCache
end

Note

If data dictionaries are open, you cannot use the command Simulink.data.dictionary.cleanupWorkerCache. To identify open data dictionaries, use Simulink.data.dictionary.getOpenDictionaryPaths.

Data Concurrency Issues

Data concurrency issues refer to scenarios for which software makes simultaneous attempts to access the same file for data input or output. In Simulink®, they primarily occur as a result of the nonsequential nature of the parfor loop during simultaneous simulations of the same model. The most common incidences arise when code is generated or updated for a simulation target of a Stateflow® chart, a referenced model, or a MATLAB Function block during parallel computing. The cause, in this case, is that the software tries to access target data from multiple worker sessions at the same time. Similarly, To File blocks might cause errors by attempting to log data to the same file during parallel simulations. A third-party blockset or custom S-function might cause a data concurrency issue while simultaneously generating code or files.

A secondary cause of data concurrency is due to the unprotected access of network ports. This type of error occurs, for example, when blocks communicate with other applications during simulation using TCP/IP. One such example is the HDL Verifier™ for use with the Siemens® ModelSim™ HDL simulator.

Resolving Data Concurrency Issues

The core requirement of parfor is the independence of the different iterations of the parfor body. This restriction is not compatible with the core requirement of simulation via incremental code generation, for which the simulation target from a prior simulation is reused or updated for the current simulation. During the parallel simulation of a model that involves generating a simulation target, such as accelerator mode simulation, the software makes concurrent attempts to access (update) the simulation target. However, you can avoid such data concurrency issues by creating a temporary folder within the parfor loop and then adding several lines of MATLAB code to the loop to perform the following steps:

  1. Change the current folder to the temporary, writable folder.

  2. In the temporary folder, load the model, set parameters and input vectors, and simulate the model.

  3. Return to the original, current folder.

  4. Remove the temporary folder and temporary path.

In this manner, you avoid concurrency issues by loading and simulating the model within a separate temporary folder. Following are examples that use this method to resolve common concurrency issues.

Models with Stateflow Charts, MATLAB Function Blocks, or Model References

In this example, the model is configured to simulate in accelerator mode or contains one or more Stateflow charts, MATLAB Function blocks, or model references. Examples of such models include sf_bounce and sldemo_autotrans). For these cases, the software generates the simulation target during the initialization phase of simulation. Simulating such a model in a parfor loop causes each simulation to generate the simulation target using the same files at the same time, while the initialization phase is running on the worker sessions. As shown in this code, you can avoid such data concurrency issues by running each iteration of the parfor body in a different temporary folder.

parfor i=1:4
   cwd = pwd;
   addpath(cwd)
   tmpdir = tempname;
   mkdir(tmpdir)
   cd(tmpdir)
   load_system(mdl)
   % set the block parameters, such as filename for To File block
   set_param(someBlkInMdl,blkParamName,blkParamValue{i})
   % set the model parameters by passing them to the sim function
   out{i} = sim(mdl,mdlParamName,mdlParamValue{i});
   close_system(mdl,0);
   cd(cwd)
   rmdir(tmpdir,'s')
   rmpath(cwd)
end

You can also avoid other concurrency issues due to file I/O errors by using a temporary folder for each iteration of the parfor body.

On Windows® platforms, consider inserting the command evalin('base','clear mex'); before the command rmdir(tmpdir, 's'). This sequence closes MEX files first before calling rmdir to remove the temporary directory tmpdir.

evalin('base','clear mex');
rmdir(tmpdir,'s')

Models with To File Blocks

If you simulate a model that contains one or more To File blocks inside a parfor loop, the nonsequential nature of the loop can cause file I/O errors. To avoid such errors during parallel simulations, you can either use the temporary folder solution or run the simulation in rapid accelerator mode with the option to append a suffix to the file names specified in the To File block parameters. By providing a unique suffix for each iteration of the parfor body, you can avoid the concurrency issue. 

rtp = Simulink.BlockDiagram.buildRapidAcceleratorTarget(mdl); 
       parfor idx=1:4 
       sim(mdl,... 
           'ConcurrencyResolvingToFileSuffix',num2str(idx),... 
           'SimulationMode','rapid',... 
           'RapidAcceleratorUpToDateCheck','off'); 
        end

See Also

Topics