MATLAB Answers

working with large matrices

73 views (last 30 days)
Bram Stegeman
Bram Stegeman on 6 Mar 2019
Commented: Stephan Koschel on 17 Mar 2020
I have to work with large matrices (e.g. A = 8000 x 100.000, all non-zero values).
I want to calculate T= ((A'*A) + lamda*speye(n))\(A'*A); with lamda =e.g. 1-e-3.
I have installed 64 gb ddr and as expected I run out of memory (also when I introduce a threshold and force A to be sparse, and then create sparse(A) and sparse (A') and try to calculate T.
Are there alternative ways to calculate T without out of memory issues?

  3 Comments

David Goodmanson
David Goodmanson on 7 Mar 2019
Hello Bram,
If A is 8e3 x 1e5 then T is going to be 1e5 x 1e5, is that correct?
Bram Stegeman
Bram Stegeman on 7 Mar 2019
Hello David,
That is correct. 1e5 x 1e5 could already results in memory issues is my experience.
Forget to mension that I'm only interested in the diagonal values of T.
Stephan Koschel
Stephan Koschel on 17 Mar 2020
You are only interested in the diagonals of a matrix multiplication?
I would implement an iteration over the diagonal elements and load the corresponding columns and rows from the two matrices. The entry on the diagonal becomes something like sum(current_row .* current_col)
The iteration could slow down the process, but you only need to load two vectors into memory.

Sign in to comment.

Answers (1)

Munish Raj
Munish Raj on 18 Mar 2019
Hello Bram,
You could look at implementing Tall Arrays.
The documentation link can be found here.

  0 Comments

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!