How to control which GPUs and CPUs get which tasks during multiple calls to trainNetwork?

I am working on a machine with a number of CPU cores (40) and a number of GPUs (4). I need to train a large number of shallow LSTM neural networks (~500,000), and would like to use my compute resources as efficiently as possible.

Here are the options I've come up with:

1) parpool('local') gives 40 workers max, which are the number of CPU cores available. Apparently parpool('local') does not provide access to the GPUs - is this correct? I can then use spmd to launch separate instances of trainNetwork across individual CPUs on my machine, and this runs 40 such instances at a time.

I have three questions about this:

First, is there a way to use both the GPUs and CPUs as separate laboratories (i.e., with different labindex values) in my spmd loop? Why do I not have a total of 44 avaialble workers from parpool?

Second, is there a way to assign more than one CPU to a particular lab, for example, could I divide my 40 cores up into 8 gorups of 5 and deploy a separate instance of trainNetwork to each of the 8 groups?

Third, given that I am using LSTMs, my 'ExecutionEnvironment' options are 'gpu', 'cpu', and 'auto', but it apears that the 'cpu' option uses more than one cpu at a time, because the timing for each task increases by a factor of about ~6 when I use spdm vs. only running one instance of trainNetwork (with 'ExecutionEnvironment' = 'cpu') at a time - this leads me to belive that when I run a single instance of trainNetowrk with 'ExecutionEnvironment' = 'cpu' it uses more than one CPU core. Is this correct?

2) I can access the GPUs individually using gpuDevice, and I can run 4 instances of trainNetwork simultaneously on my 4 GPUs. This works well, with effectively linear spedup as compared to only using one GPU at a time, but apparently does not take advantage of my CPUs.

Ideally, I'd lke a way to (1) test scaling across multiple CPUs for my partiucalr trainNetwork problem, and (2) a way to run multiple parallel instances of trainNetwork that use all of my hardware. Ideally, the best option seems to be to let the GPUs each take a number of the trainNetwork instances in parallel, and then to deploy groups of CPUs (with optimal size currently unknown) to handle a number of the trainNetwork instances.

Is there a way to do this?

Thank you,

Grey

Answer 1

[Joss Knight]

The computation on the GPU is so much faster than on the CPU for a typical Deep Learning example that there are only disadvantages to getting the CPU cores involved for the most intensive parts of the computation. Of course the CPU is being used, for all the MATLAB business logic, but that is generally low overhead and not suitable for GPU execution.

When you train on the CPU only the heavy computation is heavily vectorized and multithreaded, so there is a good chance that moving to parallel execution won't give much of an additional advantage. Parallel execution for multi-cpu comes more into its own when you go multi-node, i.e. have a cluster of multiple machines.

You can control how much multi-threading MATLAB does using maxNumCompThreads. You can run this inside parfevalOnAll, perhaps, to set the multi-threading level on your pool before training. That way you may find a good balance between numbers of MATLABs and number of threads for your particular network. You may indeed find that for your network, there is an ideal pool size for which training in parallel is effective even on a single machine.

[Grey Nearing]

Hi Josh, thanks for your answer. I'm not doing deep learning, and the Matlab's LSTMs do not have the maxNumCompThreads option, as far as I can tell. LSTM training appears to be restricted to a single CPU or single GPU. A single GPU is ~10x faster for this particualr job than a single CPU.

My issue is that I want to train a lot of (independent) shallow LSTM networks, and to use all of my resources. Right now, I've grabbed all (4) of my GPUs and used spmd to train four LSTMs at a time by using gpuDevice. Similarly, I'm running a separate script under a different instance of matlab with parpool to grab all (40) of my CPUs, and I've split my (~500K) training jobs between these two scripts; parpool is apparently not grabbing my GPUs, but gpuDevice does.

Is there a way to put my GPUs and CPUs all in the same parpool, and then use spmd or similar to push a number of independent jobs to my different resources?

[Joss Knight]

So, are you using batch or parfeval to run your jobs? In parfeval, you can retrieve the task ID like this:

t = getCurrentTask;
t.ID;

This identifies the worker, so you can use it to ensure the same workers are always using the GPU. You might start your parfeval function with something like

t = getCurrentTask;
useGpu = t.ID < 5;
if useGpu
    gpuDevice(t.ID);
    % Do GPU work
else
    % Do CPU work
end

In fact, you don't actually need to select the device if this is the way you're doing it because workers 1 to 4 will automatically be assigned GPUs 1 to 4.

maxNumCompThreads is a global utility for setting the behaviour of the running MATLAB. You can use it inside your parfeval function to control the threading behaviour of your CPU code. You can also control this by editing your cluster preferences. The default behaviour for a pool is 1 thread per worker; you could increase this while decreasing the pool size to change the balance between MATLAB processes and threads and see if there is an optimal load for your problem.

[Grey Nearing]

Hi Joss (sorry for calling you Josh earlier), thanks again for this help. I am currently implementing a parfeval implmentation - this could help.

You said that workers 1:4 should be the GPU, but when I start parpool, it only allows me to access 40 workers. I have 40 CPUs and 4 GPUs - why is it only allowing me to grab 40 workers with parpool? Is there another way to open a parallel environment.

Here is the message:

>> parpool('local',44)

Starting parallel pool (parpool) using the 'local' profile ...

Error using parpool (line 113)

You requested a minimum of 44 workers, but the cluster "local" has the NumWorkers property set to allow a maximum of 40 workers. To run a communicating job on more workers than this (up to a maximum of 512 for the Local cluster), increase the value of the NumWorkers property for the cluster. The default value of NumWorkers for a Local cluster is the number of cores on the local machine.

Any ideas?

Thanks again.

[Joss Knight]

The answers to these questions are in the documentation for the features you are using. I dare say you may benefit from reading around a bit to get the feel for things like parpool, local clusters, Parallel Preferences and the Cluster Profile Manager.

The default local cluster settings are to give you one worker per physical core and not allow you more; apparently you have 40 physical cores (lucky you!). You also get one compute thread per worker. We do this to prevent users getting a terrible experience with parallel language and blaming us. However, it's perfectly possible for you to create a new local cluster profile (for which I suggest you consult the documentation) with no limits on the maximum number of workers or number of compute threads.

The GPU is not like a CPU, think of it like a maths coprocessor. Under the hood, MATLAB will operate on gpuArrays by launching GPU kernels, one command at a time. You can't use the GPU without also using the CPU (to launch kernels, run the MATLAB desktop and interpreter, do business logic etc) so it makes sense to not oversubscribe your CPU cores by pretending that your CPU is entirely idle while the GPU does work. If you want to anyway, create a new Cluster Profile and give it a go.

[Grey Nearing]

Here is the part that answers my question:

"You can't use the GPU without also using the CPU ..."

Thanks. In hindsight it's obvious.