Append new values from a streaming time series to a saved .mat file?

Hi.

I have data continuously coming in to matlab every few seconds that I want to save to a file with as little delay as possible so that other computers can quickly access the data.

The way I have it set up right now the data gets stored in a matrix and saves it to the computer and then the loop starts over and the matrix gets one value bigger and saves it all over again and so on and so on. This works fine until I start reaching a size of a few thousand values, naturally it then starts to slow down since its writing bigger and bigger files to the hard drive. And after a while the lags are so great that the approach becomes useless. I’m assuming the save command overwrites all the old data every loop without acknowledging the duplicates and that’s what’s taking so long. Is there any way to get around this?

How is logging data with matlab usually done, is it possible to make the save command only focus on adding the last arriving value to an existing .mat file without overwriting the whole thing every iteration?

Answer 1

[Jan]

The problem does not only concern the writing to the hard disk, but creating a large array which grows iteratively consumes a lot of resources also:

v = [];
for k = 1:1e6
  v(k) = 1;
end

This does not only write 1e6 ones to the memory. In the first step [1] is created, in the 2nd step [0,0] is created at first, than [1] is copied to the first element, the original array is freed and the next 1 is inserted as last element. Finally you do not reserve and write the memory for 1e6 elements, but for sum(1:1e6) elements: > 250 GB!

A pre-allocation solves the problem:

v = zeros(1, 1e6);
for k = 1:1e6
  v(k) = 1;
end

Now there is no re-allocation and data copy in each iteration. And similar problems concerns writing to the disk.

The best solution is either to create a memory mapped file or to open a binary file and append the new data:

fid = fopen('DataFile.dat', 'w');
if fid == -1, error('Cannot create file.'); end
for k = 1:1000
   newData = rand(1, 1000);
   fwrite(fid, newData, 'double');
end
fclose(fid);

[PetterS]

Thanks, but when I try your fwrite(fid, newData, 'double'); command with my data the .dat file I get doesn’t look right, instead of seeing my numbers its full of gibberish symbols like:

p7/ ÙtB p7/ ÙtB p7/ ÙtB l/ ÙtB l/ ÙtB l/ ÙtB l/ ÙtB l/ ÙtB l/ ÙtB l/ ÙtB

Do I need to specify the formatting or something somehow for matlab to write it correctly?

And is it not possible to do the same thing but keep the data as a matlab format (.mat) or does it need to be .dat?

[Thorsten]

It's binary format. Try fread to read from the file.

[PetterS]

Ok thanks, fread managed to import them as numbers but I’m still not getting the formatting. If I for example send a 10x3 matrix to the .dat file with fwrite and then open it with fread it is now a 240x1 variable and it’s difficult to make sense of what has happened to the data.

And I don’t think it’s just the formatting, the numbers themselves seem different.

[want2know]

Hi PetterS, may I know how did solve your problem eventually? I followed this thread and have the same issue...