MATLAB Answers

How do I write a 3D matrix to a file, and then read it back in?

3 views (last 30 days)
Ralph Horsley
Ralph Horsley on 30 Sep 2020
Commented: Ameer Hamza on 1 Oct 2020
I'm running 2D wavefield simulations over time and need to store the wavefield at each time-step. The whole wavefield matricies over time range up to 4200 x 2000 x 2000 in size, so quite large. Storing the whole wavefield over time in matlab is taking up all of its memory, so I'm resorting to writing them to files and then reading them back in when needed for calculations. How would I go about writing the 3D matrix to a file, then reading it back in when needed?
Any help is greatly appreciated, thank you!

  7 Comments

Show 4 older comments
Ralph Horsley
Ralph Horsley on 30 Sep 2020
Oh that's brilliant, thank you! Does it store the values to their original precision?
Ameer Hamza
Ameer Hamza on 30 Sep 2020
Ralph, .mat is native file format to save data in MATLAB. It store the values in exactly same precision, as stored in MATLAB workspace.

Sign in to comment.

Answers (1)

Michael Croucher
Michael Croucher on 30 Sep 2020
I haven't played with the matfile function in this way before so tried a little experiment that may also serve to help you use this functionality. No idea if this is the optimal way of using it though!
If your matrices are dense and double precision then they are indeed large! Each double is 8 bytes so the matrix is
4200*2000*2000*8/(1024*1024*1024) = 125.17 % Gigabytes
Let's do a demo on a smaller version of your problem. Say, 10 pages. Here's one way of creating some data one page at a time and then saving it to the matfile.
m = matfile('myFile3.mat');
x = 4200;y=2000; %size of each matrix
N=10; % Number of pages
m.data = double.empty(x,y,0); %Initialise an empty matrix in the matfile
for i=N:-1:1
m.data(:,:,i) = rand(x,y); %There's only ever one page of the 3D matrix in memory at any one time
end
You can subsequently get any submatrix of the data without loading it all by doing, for example,
m.data(1:5,1:5,3); % the first 5 x 5 sub matrix of page 3
You may notice that I am creating the matrix inside the .mat file in reverse order - starting from N=10 and going backwards. This is because I initially tried this
m = matfile('myFile3.mat');
x = 4200;y=2000; %size of each matrix
N=10; % Number of pages
m.data = double.empty(x,y,0);
for i=1:N
m.data(:,:,i) = rand(x,y);
end
and received the error message.
Error using mymatfile (line 7)
Variable 'data' has 2 dimensions in the file, this does not match the 3 dimensions in the
indexing subscripts.
Which I wasn't expecting! It seems that setting m.data(:,:,1) makes m.data two dimensional. At this point I wonder if there is a better way to do this.

  3 Comments

Ralph Horsley
Ralph Horsley on 30 Sep 2020
Crikey that is an interesting method for it! I am using my universities bespoke computer lab for these simulations and the computers and servers are designed for handling and storing these large amounts of data so I may keep it simple with just storing them in mat files (though perhaps in single rather than double format), thank you for your effort though! One thing, I thought each double was 8 bits, not 8 bytes, I may be wrong though.
Michael Croucher
Michael Croucher on 30 Sep 2020
Definitely 8 bytes in a double. We can ask MATLAB:
>> a=1.0
a =
1
>> whos
Name Size Bytes Class Attributes
a 1x1 8 double

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!