In a Matlab script, I have to multiply a large sparse matrix with a full matrix, as follows:
A1 = B * C.
B is a full matrix, its size is 5000-by-4800 double. C is a sparse matrix, its size is 4800-by-3E5 sparse double, and its nnz/numel = 2.08E-4, so it’s very sparse.
My desktop computer crashed at this line of code. I thought it was because matrix C is too large, so I “broke down” matrix C along its columns, and let them multiply matrix B one by one, such as follows:
The computer did not crash this time, it was able to calculate the multiplication and provide matrix A1. However, it was slow (elapsed time was about 10 seconds), and the “Physical Memory Usage” in the Task Manager was alarmingly approaching the maximum.
Immediately following this “for” loop, I have one more line of code, which is a simple summation of two matrices:
A = A1 + A2.
Matrix A1 (a full matrix) is obtained from above. Matrix A2 is a sparse matrix of the same size (5000-by-3E5), it is very sparse.
As I mentioned, the computer did not crash calculating the “for” loop. But it crashed at this summation line of code afterwards. Could it be because too much “Physical Memory Usage” was used up already prior to the summation?
In conclusion, the whole thing is a mess, and I appreciate any help to be able to calculate these large matrix multiplication and then summation. Or could it be that the matrix C is simply too large and there’s no way to solve it?
Update: I just tried something else: Without running any code, I entered in the Command window: A1 = zeros(5000, 3E5). This simply creates a full matrix of zeros, and I see that the “Physical Memory Usage” in the Task Manager immediately goes up to about 14GB, approaching the maximum (16GB).
Does this mean a full matrix that has 3E5 number of columns is just too big for the computer to handle, and there's nothing I can do to solve it?