matfile operating very slowly: how to improve performance?
15 views (last 30 days)
Show older comments
Hello,
I am trying to use matfile to save some variables for a behavioral experiment (including feedback and what I presented). However, when I save these pre-defined variables to a .mat file and later on access the matrix to fill in the responses for these variables, performance is very bad. It is acceptable for the first index, but saving goes slower and slower later on to the point that it takes a few seconds...
I'd still like to save after every trial... What should I do to increase performance?
Noise = cell(maxtrials, maxsessions);
Signal = cell(maxtrials, maxsessions);
Trigger = cell(maxtrials, maxsessions);
TrialNr = (zeros(maxtrials, maxsessions));
GainSignal = zeros(maxtrials, maxsessions);
GainNoise = zeros(maxtrials, maxsessions);
SNR = zeros(maxtrials, maxsessions);
Reversal = zeros(maxtrials, maxsessions);
Correct = zeros(maxtrials, maxsessions);
MouseClick = cell(maxtrials, maxsessions);
CorrectID = cell(maxtrials, maxsessions);
SubjectAnswerID = cell(maxtrials, maxsessions);
SubjectAnswer = cell(maxtrials, maxsessions);
save_name = 'test.mat' % edit
resultsfile = matfile(save_name) %edit
resultsfile.Properties.Writable = true; %edit
save(save_name, 'Exp', 'Noise', 'Signal', 'Trigger', 'TrialNr', 'GainSignal', 'GainNoise', 'SNR',...
'Reversal', 'Correct', 'MouseClick', 'CorrectID', 'SubjectAnswerID', 'SubjectAnswer', '-v7.3');
%example: this becomes slow for later trials and sessions
resultsfile.TrialNr(trialnum, sesnum) = trialnum;
0 Comments
Answers (2)
Jan
on 16 Jan 2019
Edited: Jan
on 16 Jan 2019
You explain, that the use of the matfile command is slow, but the shown code does not contain this command at all. If you access a MAT file to append data, remember that Matlab has to import the existing data at first and due to the compression triggered by the -v7.3 format the data must be decompressed at first. Then Matlab appends the new data in the RAM and inserts it into the MAT file again after compressing it. Therefore it might be required to move parts of the .mat file on the disk to get enough space to insert the grown data. Of course this takes a lot of time, when the array grows.
If you want a faster processing, use binary files, which allows to add new data by appending them at the end of the existing file.
1 Comment
Walter Roberson
on 16 Jan 2019
7 of the variables are cell arrays. Although you are preallocating their general shape, the storage for the content is not preallocated and is potentially variable sized.
If you were to use fixed sized arrays, then potentially performance would be higher. Using fixed sized arrays would also permit you to switch to a binary file format. You might find it appropriate to use memory mapping of a struct.
Tijmen Wartenberg
on 16 Jan 2019
1 Comment
Walter Roberson
on 16 Jan 2019
The storage arrangement for -v7.3 is very different, using a modified HDF5, which has facilities for accessing and modifying and adding (chunks of?) substructures without having to rewrite the entire variable.
Compression is on by default in -v7.3, but in R2017a a -nocompression flag was added for save()
See Also
Categories
Find more on Workspace Variables and MAT-Files in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!