Main Content

Program Independent Jobs on a Local Cluster

Create and Run Jobs with a Local Cluster

Some jobs require more control than the functionality offered by high-level constructs like spmd and parfor. In such cases, you have to program all the steps for creating and running the job. Using the local cluster (or local scheduler) on your machine lets you create and test your jobs without using the resources of your network cluster. Distributing tasks to workers that are all running on your client machine does not offer any performance enhancement. Therefore this feature is provided primarily for code development, testing, and debugging.

Note

Workers running in a local cluster on a Microsoft® Windows® operating system can display Simulink® graphics and the output from certain functions such as uigetfile and uigetdir. (With other platforms or schedulers, workers cannot display any graphical output.) This behavior is subject to removal in a future release.

This section details the steps of a typical programming session with Parallel Computing Toolbox™ software using a local cluster:

The objects used by the client session to interact with the cluster are only references to data in the cluster job storage location, not in the client session. After jobs and tasks are created, you can close your client session and restart it, and your job still resides in the storage location. You can find existing jobs using the findJob function or the Jobs property of the cluster object.

Create a Cluster Object

You use the parcluster function to create an object in your local MATLAB® session representing the local scheduler.

c = parcluster('local');

Create a Job

You create a job with the createJob function. This statement creates a job in the cluster job storage location and creates the job object job1 in the client session. If you omit the semicolon at the end of the command, it displays some information about the job.

job1 = createJob(c)
 Job

    Properties: 

                   ID: 1
                 Type: independent
             Username: mylogin
                State: pending
       SubmitDateTime: 
        StartDateTime: 
     RunningDuration: 0 days 0h 0m 0s
           NumThreads: 1

      AutoAttachFiles: true
  Auto Attached Files: List files
        AttachedFiles: {}
    AutoAddClientPath: false
      AdditionalPaths: {}

    Associated Tasks: 

       Number Pending: 0
       Number Running: 0
      Number Finished: 0
    Task ID of Errors: []
  Task ID of Warnings: []

The State property of the job is pending. This means that the job has not yet been submitted (queued) for running, so you can now add tasks to it.

The scheduler display now indicates the existence of your job, which is the pending one, as appears in this partial listing:

c
 Local Cluster

    Properties: 

                   Profile: local
                  Modified: false
                      Host: myhost
                NumWorkers: 6
                NumThreads: 1

        JobStorageLocation: C:\Users\mylogin\AppData\Roaming\MathWorks\MATLAB\local_cluster_jobs\R2021b
   RequiresOnlineLicensing: false

    Associated Jobs: 

            Number Pending: 1
             Number Queued: 0
            Number Running: 0
           Number Finished: 0

Create Tasks

After you have created your job, you can create tasks for the job using the createTask function. Tasks define the functions to be evaluated by the workers during the running of the job. Often, the tasks of a job are all identical. In this example, five tasks each generate a 3-by-3 matrix of random numbers.

createTask(job1, @rand, 1, {{3,3} {3,3} {3,3} {3,3} {3,3}});

The Tasks property of job1 is now a 5-by-1 matrix of task objects.

job1.Tasks
 5x1 Task array:
 
         ID        State              FinishDateTime  Function  Errors  Warnings
       -------------------------------------------------------------------------
    1     1      pending                                  rand       0         0
    2     2      pending                                  rand       0         0
    3     3      pending                                  rand       0         0
    4     4      pending                                  rand       0         0
    5     5      pending                                  rand       0         0

Submit a Job to the Cluster

To run your job and have its tasks evaluated, you submit the job to the cluster with the submit function.

submit(job1)

The local scheduler starts the workers on your machine, and distributes the tasks of job1 to these workers for evaluation.

Fetch the Job Results

The results of each task evaluation are stored in the task object OutputArguments property as a cell array. After waiting for the job to complete, use the function fetchOutputs to retrieve the results from all the tasks in the job.

wait(job1)
results = fetchOutputs(job1);

Display the results from each task.

results{1:5}
ans =

    0.1349    0.5332    0.2621
    0.6744    0.1150    0.9625
    0.9301    0.6540    0.8972


ans =

    0.6383    0.6509    0.4429
    0.5195    0.3018    0.3972
    0.1398    0.7101    0.7996


ans =

    0.9730    0.2934    0.6071
    0.7104    0.1558    0.5349
    0.3614    0.3421    0.4118


ans =

    0.3241    0.9401    0.1897
    0.0078    0.3231    0.3685
    0.9383    0.3569    0.5250


ans =

    0.4716    0.6667    0.7993
    0.5674    0.6959    0.9165
    0.3813    0.8325    0.8324

After the job is complete, you can repeat the commands to examine the updated status of the cluster, job, and task objects:

c
job1
job1.Tasks

Local Cluster Behavior

The local scheduler runs in the MATLAB client session, so you do not have to start any separate scheduler or MATLAB Job Scheduler process for the local scheduler. When you submit a job to the local cluster, the scheduler starts a MATLAB worker for each task in the job. You can do this for as many workers as allowed by the local profile. If your job has more tasks than allowed workers, the scheduler waits for one of the current tasks to complete before starting another MATLAB worker to evaluate the next task. You can modify the number of allowed workers in the local cluster profile. If not specified, the default is to run only as many workers as computational cores on the machine.

The local cluster has no interaction with any other scheduler or MATLAB Job Scheduler, nor with any other workers that can also be running on your client machine under the mjs service. Multiple MATLAB sessions on your computer can each start its own local scheduler with its own workers, but these groups do not interact with each other.

When you end your MATLAB client session, its local scheduler and any workers that happen to be running also stop immediately.