is it faster to load a big file (198MB) once and pass it onto a function in a parfor loop or let the function in parfor loop to load it every iteration?

Dear All,

is it faster to load a big .mat file (198MB) once and pass it onto a function in a parfor loop or let the function in parfor loop to load it every iteration? My hunch is that to load it once (which take about 10min) and let it pass to the function. But I am not sure. Also, are there faster ways to load .mat file? Currently, I am simply using the "load" function.

Another question is that will the script run faster if I put all the codes in a single m file as compared to separating them into different functions (also m files). The readability will improve but I am not sure if the computation efficiency is significantly impacted. Thanks! best, Gene

[Jason Ross]

Are you running everything locally on one machine, or are your workers distributed among several machines? Do you plan on running this on a cluster anytime soon? Is there a shared file system if you are planning on doing so?

Answer 1

[Jason Ross]

You might find Edric's FEX submission worth reviewing:

http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper

Answer 2

[Dan K]

A few things...

1 it should not take minutes to load a file that's only 200MB, so you may want to look at how you're loading it.

2. Yes, it is faster to load once then pass it around, since RAM is faster than a HD.

3. It doesn't matter if it's one file or many, but.... You'll get much more benefit by converting your script into a function