How can I make data files available to my code running on MATLAB Distributed Computing Server?

10 views (last 30 days)
I have a MATLAB program which uses some data files. These files are currently stored on my local computer. I can use these data files without issue when running parallel code on my local machine, but when I try to run on my MATLAB Distributed Computer Server cluster, I receive errors saying that the files cannot be found.
How can I make these data files available to my code running on the cluster?

Accepted Answer

MathWorks Support Team
MathWorks Support Team on 4 Jan 2022
Edited: MathWorks Support Team on 4 Jan 2022
There are three ways to make local data files available to workers on a cluster:
​1) Create a job for your computation, and attach files to the job. This option does not require infrastructure changes but will not scale well if you have many workers, large files, or a large number of files. The following example creates a job with attached files, adds a task, and submits the job. The code will need to be changed to refer to just the filename included in 'AttachedFiles', instead of the path to the file on the local machine.\n
c = parcluster('myRemoteClusterProfile');
j = batch(c, @myFunc, 1, {10,10}, 'AttachedFiles', {'myData.csv'}); % myFunc has 1 output argument and two inputs
Refer to the following documentation for more information about creating jobs for a cluster:
2) Start a parallel pool, and attach files to the job. This option is very similar to (1), but files will be attached to a parallel pool instead of a job. The files will remain on the workers while the pool is open. The same considerations apply to this approach as (1). Example:
c = parcluster('myRemoteClusterProfile');
poolobj = parpool(c);
addAttachedFiles(poolobj, {'file1.mat'});
Refer to the following documentation for more information about attaching files to a parallel pool:
3) Place the data in a networked file share which the worker machines can access
. This option may require some infrastructure changes depending on your network, however this option scales better for large files and many workers. Your code would need to use the path to the data at the network location instead of the path to the data on the local hard drive.
  1 Comment
Raymond Norris
Raymond Norris on 4 Jan 2022
You'll want to use the UNC path, not mapped drive. Permission issues makes it sound like there's a conflict with security levels of MJS. Contact MathWorks Technical Support (support@mathworks.com) for more help.

Sign in to comment.

More Answers (0)

Categories

Find more on MATLAB Parallel Server in Help Center and File Exchange

Products


Release

R2016b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!