Best way to save big matrices

11 views (last 30 days)
Ingo
Ingo on 29 Dec 2011
I'm looking for the best way to save big (both full and sparse) matrices in files. What I need to be able to do is:
- Read only certain entries from the file (certain matrix entries, that is)
- Create a sparse matrix in MATLAB from this file without creating a full matrix first (assuming the file contains a sparse matrix, of course)
The reason is quite simple: The nonsparse matrices will be too big to be stored into memory as a full matrix.
I read about the CDF file format which I'm not yet familiar with, so I don't know if it's possible to create a sparse matrix in Matlab from a CDF File?
  1 Comment
Ingo
Ingo on 29 Dec 2011
A little elaboration:
Given are n data points and now I want to calculate the (n x n) matrix containing the pairwise distance between all points. This is a full matrix, too big to store it in memory.
The next step would be to make this matrix sparse by, for example, ignoring all values less than a certain treshold. This sparse matrix can be stored in memory.

Sign in to comment.

Accepted Answer

the cyclist
the cyclist on 29 Dec 2011
The most recent versions of MATLAB have the matfile() command, which I believe does at least some of what you want.
  3 Comments
the cyclist
the cyclist on 29 Dec 2011
If you do find it works out for you, please "accept" the solution, which may help future users with similar problems.
Ingo
Ingo on 30 Dec 2011
Hi. Sorry for being late on that!
Yes, matfile works for what I intended to do. However, for my purposes it is just way too slow. I'll have to stick with smaller data sets and try to create the resulting sparse matrices without ever calculating full matrices.
However, matfile() still does solve the problem.
Thanks again and Greetings
Ingo

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!