Cluster with 8 workers runs only one job at a time

I am using a local cluster (parcluster) to run 6 jobs. When I examine my cluster, I see that it reports 8 workers. Since I'm submitting the 6 jobs simulataneously, I would expect that all six would start running right away. Instead, the jobs are only executed one at a time with the others remaining in a 'queued' state. Is there a way to run more than one job at a time, or (best case) run all jobs simultaneously?

[Edric Ellis]

How are you submitting these jobs? I just tried this using the BATCH command, and saw this:

>> c = parcluster('local');
>> for idx = 1:6, batch(c, 'pause(10)'); end
>> c.Jobs

ans = 

   6x1 Job array:

           ID           Type        State       FinishTime  Username  Tasks
         ------------------------------------------------------------------
      1     1    independent      running                     eellis      1
      2     2    independent      running                     eellis      1
      3     3    independent      running                     eellis      1
      4     4    independent      running                     eellis      1
      5     5    independent      running                     eellis      1
      6     6    independent      running                     eellis      1

[Stephen]

The jobs are submitted as a group of 20 tasks using createTask, as follows:

myCluster = parcluster;
for i = 1:6
  workerTable{i,1} = createJob(myCluster);
  for j = 1:20
    createTask(workerTable{i,1},@lengthyFxn,7,{inputArgs});
  end
  submit(workerTable{i,1});
end

Answer 1

[Sean de Wolski]

The tasks are not running serially as you expect but are running in parallel and are thus using all of the resources available. This is why the next job has to wait until the previous has finished running all of its tasks.

[Stephen]

So am I understanding correctly that my 8 workers are getting individual tasks, not jobs?

[Sean de Wolski]

Yes.

[Stephen]

How could I increase the number of workers in the parpool? Would increasing the number of workers help the background processes to finish sooner or is that capped by system resources?