Perfomance Loss of Matrix-Vector Multilplication on GPU with Array Indexing

Hi,

I have a large matrix A and a vector B. I want to do a partial multiplication on GPU using array indexing but the peformance is much lower than doing a full A*B. Below is a simple example of what I am trying to do:

A = rand(20000,'gpuArray');
B = rand(20000,1,'gpuArray');
C = A(8001:18000,1:end)*B;

GPU Device: Tesla V100

MATLAB 2020a

Any suggestion on how to improve the performance? Thank you.

Answer 1

[Edric Ellis]

Unfortunately, the expression A(8001:18000,:) requires a strided memory copy. Matrices in MATLAB (even on the GPU) are stored in column-major format, so picking out only certain rows is much less efficient than picking out only certain columns.

There's a trick you can use though that takes advantage of the fact that gpuArray matrix multiplication is optimised for the transposed-times case. Try instead pre-transposing A (this is relatively expensive, but perhaps you can do it only once) and then doing:

A(:, 8001:18000).' * B;

This uses the much-faster indexing pattern, and is about ~2x faster on my GPU.

[Afshin Ahmadi]

Thank you, Edric. I tried your solution but the execution time increased instead! Here is my code:

clc;
clear;
profile clear;
gpuDevice(1);
A = gpuArray.rand(5000);
B = gpuArray.rand(5000,1);
At = A';
profile on
C1 = gather(At(:,500:2000).'*B);
C2 = gather(A(500:2000,:)*B);
profile off
profile viewer

Can you please double check?

Actually I am thinking of using the external cuBLAS library for this purpose. I was able to compile and run the example in https://www.mathworks.com/help/coder/ug/use-custom-c-code.html. To allow selective row-block multiplication, I can point to the memory address of the first element in desired row (equivalent in C is &d_A[start_address]) and change m,n values accordignly. However, I am struggling to implement this in the above MATLAB example. Can you please tell me what to change in passing the d_A array in the following piece of code for this purpose? Thank you ver much in advance.

% Multiply matrices on GPU 
                 ret = coder.ceval('cublasDgemm', handle, ...
                    TRANSA,TRANSA,m,n,k, ...
                    coder.rref(alpha),...
                    d_A,m, ...
                    d_B,k, ...
                    coder.rref(beta),...
                    d_C,k);

[Edric Ellis]

While the MATLAB profiler attempts to gather realistic times for GPU operations, in practice it's much better to use gputimeit - and also not time time the gather operation. I modified your example slightly to the following:

clc;
clear;
gpuDevice(1);
A = gpuArray.rand(5000);
B = gpuArray.rand(5000,1);
At = A.';
t1 = gputimeit(@() At(:,500:2000).'*B)
t2 = gputimeit(@() A(500:2000,:)*B)

And got the result:

t1 =
   5.2723e-04
t2 =
   0.0015

(This is on my relatively old Tesla K20c)

I'm afraid I don't know much about the coder way of doing things, so I'm not sure how you would offset the pointer into A there.

[Afshin Ahmadi]

Interestingly, your solution is taking twice more time compared with the normal multiplication on Tesla V100! I guess they are different in the architecture. Thank you tho.

[Edric Ellis]

Strange, I just tried on a WIN64 machine here with a V100, and got the following result:

t1 =
   1.6677e-04
t2 =
   4.4944e-04

(This was using R2020a).

[Afshin Ahmadi]

I tried again and it seems your solution is quite fast when the block size is small, which is exactly what I need. Thank you so much for the help! I will just include some information here for the people who are interested in doing the same thing.

A = gpuArray.rand(20000);
B = gpuArray.rand(20000,1);
At = A.';
t1 = gputimeit(@() At(:,500:2000).'*B)
t2 = gputimeit(@() At(:,500:5000).'*B)
t3 = gputimeit(@() At(:,500:10000).'*B)
t4 = gputimeit(@() A(500:2000,:)*B)
t5 = gputimeit(@() A(500:5000,:)*B)
t6 = gputimeit(@() A(500:10000,:)*B)
t7 = gputimeit(@() A*B)

Execution time:

t1 =   4.4423e-04

t2 =    0.0010

t3 =    0.0020

t4 =    0.0035

t5 =    0.0051

t6 =    0.0076

t7 =    0.0044

(MATLAB R2020a, Tesla V100, Linux)