big matrix-out of

Hi, I have a problem with matrix dimensions. In a database of 88,000 observations, I should compute the distances among all of them, thus I need a matrix of 88,000 x 88,000. Unfortunatly, MATLAB does not allow me to do this computation since it runs out of memory. How can I solve this problem? Thanks

Answer 1

[Azzi Abdelmalek]

http://www.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html

Answer 2

[Jan]

If you need a 88'000x88'000 matrix, you require 62GB of free RAM, when you store the values as doubles. The RAM must be available in a contigious block, so better run the code on a machine with 120GB RAM.

If you do not have access to such a computer, you cannot store this matrix, even if you need it. There is no magic trick. But some strategies: Do you need double precision or is single sufficient? You can exploit the symmetry of the distance matrix, because half of the information is redundant. Then you can reduce the required resources to about 30GB. If this is still more RAM than you have, ask the person who told you that you need this computations for buying more RAM.

[Steven Lord]

If you want to perform some operation that requires making a copy, your machine better have 2 or even 3 times as much memory as is required to store the original matrix, and you'd better not do anything to fragment that memory.

Alternately, I think your best solution, since it's unlikely you need all 62 GB of results at once, is to break the problem into chunks. Find the distances from the first 1000 points to each of the 88000 points, process those distances selecting only the few you need or want to process further, and repeat 88 times.