Assigning gpuArrays to different graphics cards
Show older comments
In the example below I use a parfor loop to assign a different 256x256x256 (random k-space) matrix to each of my 2 GPUs. Theoretically, I can then process these matrices in parallel on the 2 GPUs (here I've done an ifftn). The problem is that the parfor loop is very slow (presumably overhead related). This operation takes 6.5 seconds on my machine. If I replace the 'parfor' with a simple 'for' (and run 1 GPU sequentially), this operation takes 0.2 seconds.
Is there an easy (fast) way to assign a gpuArray to a specific graphics card - such that future operations on this gpuArray will use the specified graphics card? Rather than using parfor, I would prefer to simply use asynchronous CUDA kernels (invoking one directly following the other in Matlab code) to run both GPUs in parallel.
Dims = [256,256,256,2];
Kspace = complex(rand(Dims,'single'),rand(Dims,'single'));
Image = gpuArray(complex(zeros(Dims,'single')));
tic
parfor n = 1:2
gKspace = gpuArray(Kspace(:,:,:,n));
Image(:,:,:,n) = ifftn(fftshift(gKspace));
end
toc
Accepted Answer
Joss Knight
on 22 Feb 2020
0 votes
There is no way to do what you ask. Selecting a GPU is the only way to move data there, and selecting a GPU resets all GPU data.
The issue here is the way you're sending all the data to each worker and then indexing it, this is your bottleneck (and equivalently, moving all the results back). You need to amortise this communication cost, either by doing more work inside the loop or by loading the data you need directly onto each worker without first loading it onto the client.
Presumably you have more than 2 256^3 arrays. Put another loop inside your parfor and process all those arrays together. Move the results back to the CPU to save GPU memory. Eventually the communication overhead will be irrelevant and you'll see the gain of use of both your GPUs.
6 Comments
Adam Karboski
on 15 May 2020
Edited: Adam Karboski
on 15 May 2020
That makes sense because 'Image' is common to both workers, but why do local variables get cloned?
spmd(4)
reset(gpuDevice(labindex));
pause(labindex);
blah();
end
function blah()
foo = gpuArray.zeros(1,2^29);
g = gpuDevice();
fprintf('GPU %i : %iMB free\n', g.Index, round(g.AvailableMemory/2^20));
pause(4)
end
Lab 1:
GPU 1 : 10637MB free
Lab 2:
GPU 2 : 6540MB free
Lab 3:
GPU 3 : 2440MB free
Error using untitled (line 3)
Error detected on worker 4.
Caused by:
Error using untitled>blah (line 13)
Out of memory on device. To view more detail about available memory on the
GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling
'gpuDevice(4)'.
Joss Knight
on 15 May 2020
These variables aren't cloned, they aren't even owned by the same processes. How much memory was available before you assigned the arrays? Are you using a GPU virtualization technology that shares one GPU between multiple processes?
A simple test would be to run nvidia-smi in a terminal before, and then after creating a gpuArray in a MATLAB. Does memory go down on all your devices?
Adam Karboski
on 15 May 2020
Edited: Adam Karboski
on 15 May 2020
These are physical GPUs.
(edited because I misread the nvidia-smi output) nvidia-smi is seeing each of the four workers opening each gpu, and using identical amounts of memory on each gpu.
Here is my profile:
parpool(4)
spmd(4)
gpuDevice(labindex);
end
system('nvidia-smi');
spmd(4)
pause(labindex*2);
blah();
end
function blah()
foo = gpuArray.zeros(1,2^28);
g = gpuDevice();
fprintf('GPU %i : %iMB free\n', g.Index, round(g.AvailableMemory/2^20));
system(sprintf('nvidia-smi > thread%i.txt',labindex));
pause(10)
end
>> untitled5
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 4).
ans =
ProcessPool with properties:
Connected: true
NumWorkers: 4
Cluster: local
AttachedFiles: {}
AutoAddClientPath: true
IdleTimeout: 1000 minutes (1000 minutes remaining)
SpmdEnabled: true
Fri May 15 16:16:00 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro RTX 5000 On | 00000000:1C:00.0 Off | Off |
| 33% 34C P2 40W / 230W | 1020MiB / 16114MiB | 4% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 5000 On | 00000000:1D:00.0 Off | Off |
| 33% 35C P2 46W / 230W | 1020MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Quadro RTX 5000 On | 00000000:3F:00.0 Off | Off |
| 33% 31C P2 43W / 230W | 1020MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Quadro RTX 5000 On | 00000000:40:00.0 Off | Off |
| 34% 31C P2 39W / 230W | 1020MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 0 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 0 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 0 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
+-----------------------------------------------------------------------------+
Lab 1:
GPU 1 : 13042MB free
Lab 2:
GPU 2 : 10993MB free
Lab 3:
GPU 3 : 8941MB free
Lab 4:
GPU 4 : 6889MB free
>>
Output of each thread:
Fri May 15 16:16:02 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro RTX 5000 On | 00000000:1C:00.0 Off | Off |
| 33% 35C P2 43W / 230W | 3072MiB / 16114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 5000 On | 00000000:1D:00.0 Off | Off |
| 33% 36C P2 45W / 230W | 3072MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Quadro RTX 5000 On | 00000000:3F:00.0 Off | Off |
| 33% 31C P2 44W / 230W | 3072MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Quadro RTX 5000 On | 00000000:40:00.0 Off | Off |
| 34% 32C P2 39W / 230W | 3072MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 0 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 0 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
+-----------------------------------------------------------------------------+
Fri May 15 16:16:04 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro RTX 5000 On | 00000000:1C:00.0 Off | Off |
| 33% 35C P2 40W / 230W | 5124MiB / 16114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 5000 On | 00000000:1D:00.0 Off | Off |
| 33% 36C P2 48W / 230W | 5124MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Quadro RTX 5000 On | 00000000:3F:00.0 Off | Off |
| 33% 32C P2 43W / 230W | 5124MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Quadro RTX 5000 On | 00000000:40:00.0 Off | Off |
| 34% 32C P2 39W / 230W | 5124MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 0 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
+-----------------------------------------------------------------------------+
Fri May 15 16:16:06 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro RTX 5000 On | 00000000:1C:00.0 Off | Off |
| 33% 35C P2 40W / 230W | 7176MiB / 16114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 5000 On | 00000000:1D:00.0 Off | Off |
| 33% 36C P2 46W / 230W | 7176MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Quadro RTX 5000 On | 00000000:3F:00.0 Off | Off |
| 33% 32C P2 46W / 230W | 7176MiB / 16117MiB | 3% Default |
+-------------------------------+----------------------+----------------------+
| 3 Quadro RTX 5000 On | 00000000:40:00.0 Off | Off |
| 34% 33C P2 39W / 230W | 7176MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 1 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 2 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
| 3 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 255MiB |
+-----------------------------------------------------------------------------+
Fri May 15 16:16:08 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro RTX 5000 On | 00000000:1C:00.0 Off | Off |
| 33% 35C P2 40W / 230W | 9228MiB / 16114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 5000 On | 00000000:1D:00.0 Off | Off |
| 33% 36C P2 45W / 230W | 9228MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Quadro RTX 5000 On | 00000000:3F:00.0 Off | Off |
| 33% 32C P2 44W / 230W | 9228MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Quadro RTX 5000 On | 00000000:40:00.0 Off | Off |
| 34% 33C P2 42W / 230W | 9228MiB / 16117MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 0 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 1 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 2 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27221 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27224 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27226 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
| 3 27228 C /data/opt/MATLAB/R2020a/bin/glnxa64/MATLAB 2307MiB |
+-----------------------------------------------------------------------------+
Joss Knight
on 15 May 2020
That is exceedingly odd and definitely the source of the problem. Before I send you to tech support, can you just make sure to open your pool using parpool('local',4), just in case your default parallel profile is messed up in some way.
Adam Karboski
on 15 May 2020
Same result. I also created a profile from scratch, same again.
Adam Karboski
on 19 May 2020
Solved, solution was to disable nvidia-persistenced
More Answers (0)
Categories
Find more on GPU Computing in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
