MATLAB Answers

parfeval getting stuck in for loop and blocking fetchNext

4 views (last 30 days)
Hi,
I'm trying to use parfeval and fetchNext, but I don't think I'm settling them up correctly. My parallel pool has 4 workers. I'm setting up parfeval in a for-loop, as shown in the manual, with fetchNext in a subsequent for-loop. I thought that fetchNext should be able to extract values from parfeval as they are produced, but it appears to only extract them once all iterations of the first loop are complete.
I'm practising with the code from the parfeval and fetchNext pages of the manual. When I put a breakpoint at the 'fetchNext' line of the code below, it always shows all of the FevalFuture items complete before the breakpoint is reached. If I include a "disp('par')" and "disp('fetch')" inside each loop then all the "disp('par')" are printed out first and then all the "disp('fetch')" are printed out afterwards.
Also, on this code using fetchNext is actually slower than fetchOutputs. The fetchNext version takes 73 seconds, whereas the fetchOutputs version at the bottom only takes 31 seconds.
Any help very much appreciated!
p = gcp();
tic
for idx = 1:10000
f(idx) = parfeval(@rand,1,1);
end
for idx = 1:10000
[completedIdx,value(completedIdx)] = fetchNext(f);
end
toc
p = gcp();
tic
for idx = 1:10000
f(idx) = parfeval(@rand,1,1);
end
R = fetchOutputs(f);
toc

Accepted Answer

Raymond Norris
Raymond Norris on 14 May 2021
Some thoughts/comments
  • I thought that fetchNext should be able to extract values from parfeval as they are produced, but it appears to only extract them once all iterations of the first loop are complete. When I put a breakpoint at the 'fetchNext' line of the code below, it always shows all of the FevalFuture items complete before the breakpoint is reached.
I suspect your code is finishing too quickly (just calling rand) and that by the time you get to the fetch you're done. Try the following. You ought to see most futures are queued.
p = gcp();
tic
for idx = 1:100
f(idx) = parfeval(@pause,0,10);
end
for idx = 1:100
completedIdx = fetchNext(f);
end
toc
  • If I include a "disp('par')" and "disp('fetch')" inside each loop then all the "disp('par')" are printed out first and then all the "disp('fetch')" are printed out afterwards.
This part is true. The for-loops are running in serial, so MATLAB can't fetch even the first future until it's submitted the last one in the first for loop.
  • Also, on this code using fetchNext is actually slower than fetchOutputs. The fetchNext version takes 73 seconds, whereas the fetchOutputs version at the bottom only takes 31 seconds.
That's true as well, but fetchNext and fetchOutputs serve different purposes. fetchNext allows you to get a result at a time. If you've found what you're looking for, you could cancel your remaining tasks, which can save signicant time. fetchOutputs will fetch all values, which might not be what you want to do. Take for example the following:
for idx = 1:10000
f(idx) = parfeval(@pause,1,10);
end
tic
for idx = 1:1
[completedIdx,value] = fetchNext(f);
end
toc
tic
for idx = 1:1000
f(idx) = parfeval(@pause,1,10);
end
tic
R = fetchOutputs(f);
toc
This is an extreme example, but if you found your answer on the first pass, it'd be 125x faster to call fetchNext then to call fetchOutputs.
  1 Comment
Felicity Freeman
Felicity Freeman on 15 May 2021
Thank you. That makes it much clearer, although it means I need to completely rethink what I'm trying to do!

Sign in to comment.

More Answers (0)

Products


Release

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!