MATLAB Answers

Huge number of iterations

6 views (last 30 days)
Qammar Abbas
Qammar Abbas on 20 Sep 2021
Edited: Qammar Abbas on 22 Sep 2021
Hi Community members,
I am generating chemical formulas of compounds by forming combinations of elements and storing them in text file. The total number of combinations according to my calculations come out to be 18,217,382,400 i.e. i need 18,217,382,400 number of for loop iterations. I want to do this as quicky as possible. Please suggest an efficient method for doing this. I have tried both for and parfor, they take too long. A snippet of my code is shown below. I am using 2 workers and the code has been running for more than 24 hours now. How can I improve speed?
fcn = @() fopen( sprintf( 'chem_%d.txt', labindex ), 'wt' );
w = WorkerObjWrapper( fcn, {}, @fclose );
iterations=[length(a) length(b)]; % a and b are cell arrays. Length of a is 10944 length of b is 1664600
tic
parfor ix=1:prod(iterations)
ix
[d,e]=ind2sub(iterations,ix);
fprintf(w.Value, '%s\n', strcat(a{d},b{e}));
end
toc
clear w;
  6 Comments
Rik
Rik on 21 Sep 2021
Then you should probably consider buying computation time on some sort of cluster. If you don't tell us what you want to do, we can't suggest a way to avoid some of the computational work. Things take time. Sometimes the most efficient way is to reduce the number of things.

Sign in to comment.

Answers (1)

Walter Roberson
Walter Roberson on 20 Sep 2021
fcn = @() fopen( sprintf( 'chem_%d.txt', labindex ), 'wt' );
w = WorkerObjWrapper( fcn, {}, @fclose );
% a and b are cell arrays. Length of a is 10944 length of b is 1664600
b = b(:);
tic
iterations = length(a);
parfor ix=1:iterations
outs = strjoin(strcat(a(ix), b, {newline})); %a(ix) is deliberate in case a{ix} has whitespace
fwrite(w.Value, outs);
end
toc
  4 Comments
Qammar Abbas
Qammar Abbas on 22 Sep 2021
I have tried your first code and as @Benjamin explained, indeed it is a very good solution to my problem. However, I observed that the execution time further reduces if we use 'for' instead of 'parfor' in your first code. According to my calculation, I need maximum of 2 days to generate all 18,217,382,400 combinations using for loop. I have started running the code and will get back to you with the results in 2-3 days hopefully. Meanwhile, I am trying to understand the second code you have shared. I am thankful for your help.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!