parfor loop output and input are not matched in order

I have a code segment here:

so I wish to use parfor to calculate a function called OFreduced. The OFreduced takes in 10 numbers each time as input.

I have stored all input variables in modified_geo as a matrix of 3*10, where each entire row is one input to the function. So in total I have 3 sets of input that are needed to be calculated in parfor.

The output from the function OFreduced is simply just a number, and I stored that in another array v, which should contain the 3 results from the input.

It should produce three different answers and stored in v, and I could perfectly run this in serial with no problem. However parfor gives me the same number for the three inputs (i.e. three numbers are indentical in v). It seems like I have not matched the input index to the output index, and the results are "overwriting".

Please advise how shall I modify this to "match" the outputs and the inputs in the correct order. Many thanks in advance!

% compute cost using simplified OF

modified_cost1 = zeros(3,1);

v = zeros(3,1);

% run OF with temporary variable

parfor j = 1:3

v(j) = OFreduced(modified_geo(j,:));

end

modified_cost1 = v(1,1:3);

[Rik]

Other than the odd indexing at the end (which should return an error), I don't see what you would be doing wrong here. Can you attach the function?

[Edric Ellis]

Yes, this is definitely not expected. We're going to need reproduction steps that we can execute to be able to help you here. Ideally, it would be good if you could try and come up with a "minimal" reproduction that demonstrates the problem.

Could I also ask: what type of cluster are you running on? If you're running on something other than a "local" pool, it's possible that the workers are seeing a different definition of OFreduced, and that could be causing the problem.

[Forrest Han]

@Edric EllisSo I believe that I am running on local pool on my laptop, and that

OFreduced

funtion is really a tedious function which, basically calls and runs another application in MATLAB to calculate the numerical answer given those 10 inputs.

I was aware of the "reproduction" thing at the end of the parfor loop, but I really had some trouble understanding it. I have had a look through: https://ch.mathworks.com/matlabcentral/answers/574615-parfor-unable-to-classify

Would you mind giving me some more instructions on how shall I execute the “reproduction“ steps please? Thanks a lot!

[Forrest Han]

@Rik that OFreduced function is a long function which calls and execute another application in MATLAB to produce results. That part of code looks like this

%% constrained PSO under critical mass and winglet loading
if P1 < 0
    P1 = 0;                         % if winglet mass less tha critical, continue AVL analysis
    AvlFileGeneratorreduced(x);            % generate .avl geometry file
    run("AvlRunner.m");             % run AVL in MATLAB   
    [l2dratio, load]            = ParameterExtract;          % extract l2dratio and load (TBD)
    P2                          = load - critical_load;      % winglet loading penalty   
    if P2 < 0               
        P2 = 0;             % if winglet load less than critical, output result
    else 
        
        P2 = 1;             % penalty for critical load in flight
        l2dratio = 0;

    end
else
  
    P1 = 1;                 % penalty for critical winglet mass
    l2dratio = 0;

end

The AvlRunner.m script contains a line which calls the application:

%% INPUT variables
filename = 'b737';
velocity = 10;%m/s
%Define a base file name to save 
basename = 'newdata';
%Location of avl 
avlLocation = '"C:\Users\hfpot\OneDrive -\Desktop\avl.exe"';

[Edric Ellis]

By reproduction steps - I mean code that we can execute and see the problem. It sounds like that's not going to be easily possible if you're calling an external executable.

When you execute "avl.exe" - how are you getting the results out? If it creates a file - then that's a potential problem right there. Your workers might well all be clashing trying to write the same file - unless you take steps to prevent that. One simple way to avoid that is to have the workers change to a unique directory prior to running the body of the loop, like this:

parfor j = 1:3
    oldWd = pwd();
    newDirName = tempname();
    mkdir(newDirName);
    cd(newDirName);
    ... do stuff
    cd(oldWd);
end

[Forrest Han]

@Edric Ellis This segment works! Thank you very much :)

Answer 1

[Rik]

I suspect the source of the problem is the interaction with the file system.

You should consider generating a random file name (either with randi indexing from allowed characters, or with tempname). That way you ensure you don't overwrite the file that the other process is using.

I would suggest avoiding cd if at all possible. You can use tempdir if you need the temp folder location.

[Forrest Han]

This works, thank you very much!

[Rik]

You're very much welcome.