Loading Time difference between files containing timetables and files containing numeric values

TL;DR
The who and load function are very slow when loading timetable data compared to loading numeric data. Why? Is this maybe a bug? How can I speed up these functions for timetable?
Background
I have a huge file (>1GB) containing various sensor logs of different frequencies. Right now the values and times are stored seperatly as numeric double vectors. A value vector can be mapped to a time vector through the variable name of the value vector, e.g.: "velocity_t32" can be mapped to the time vector "t32". The timestamps of the logs are irregular -> They can't be compressed into starttime+timesteps.
Now I want to switch to MATLAB timetables since they seem to be more convenient to work with. I have run some experiments to compare the two formats against each other and I found out that the matlab load and who functions run significant slower when loading a file containing timetables than loading a file containing only numeric values. I attached the live-script.
Generation of sample-files
%% Code to generate sample-files (I can't share original data, this should be close enough)
signals_as_timetables = struct();
signals_as_numeric = struct();
num_signals = 2000;
length_run_in_ms = uint64(1000000); % 1000s ~ 16min
d = uiprogressdlg(uifigure, Title='Loading Files');
drawnow;
for i = 1:num_signals
d.Value = i / num_signals;
% Mock Sample Rate
n = randi(9, 1, 1) + 1;
e = randi(3, 1, 1) - 1; % 1/3 of signals will be of sample rate 1ms
avg_sample_rate = uint64(n * realpow(10, e)); % in ms, e.g. 2ms, 40ms or 90ms
length_signal = uint64(length_run_in_ms / avg_sample_rate);
% Mock Time-Vector
times = randi(length_run_in_ms, length_signal, 1);
times = sort(times);
% Mock signal data
values = rand(length_signal, 1);
% Mock numeric Format
signals_as_numeric.("signal_XY_t"+i) = values; % e.g. signal_XY_t50
signals_as_numeric.("t"+i) = times; % e.g. signal_50(t50)
% Mock Timetable
times = milliseconds(times) + datetime(date); % Convert numeric to datetime
times.Format = 'yyyy-MM-dd HH:mm:ss.SSS Z';
signals_as_timetables.("signal_"+i) = timetable(times, values);
end
close(d);
save("timetables.mat", '-struct', 'signals_as_timetables', '-v7.3'); % file size: ca. 1.5 GB
save("numeric.mat", '-struct', 'signals_as_numeric', '-v7.3'); % file size: ca. 1.4 GB
Preview of both files:
Experiments
% Compare listing all variable names
tic
signal_names = who("-file", "timetables.mat");
toc % Elapsed time is 10.471696 seconds.
tic
signal_names = who("-file", "numeric.mat");
toc % Elapsed time is 0.163475 seconds.
As you can see there is a difference of factor 65 when loading the variable names.
% Compare loading complete file
tic
signals = load("timetables.mat");
toc % Elapsed time is 10.293126 seconds.
tic
signals = load("numeric.mat");
toc % Elapsed time is 8.330326 seconds.
Loading the complete timetable file is slower than numeric data, but the difference seems acceptable.
% Compare loading single signal
tic
signal = load("timetables.mat", "signal_1");
toc % Elapsed time is 10.212967 seconds.
tic
signal = load("numeric.mat", "signal_XY_t1");
toc % Elapsed time is 0.079693 seconds.
However, loading a single variable is super fast for the numeric data, but loading timetable data seems to be equally slow like loading the complete file. For this task the numeric data has 145 times faster loading times than timetable data.
Questions
  1. Why does this behave like this? Is there something I am missing about timetables?
  2. Is there a better way of storing this kind of data (without a timeseries-database like InfluxDB)?
  3. How can I speedup the who and load function for timetables?
I am happy about every Answer, Tip, Advice, Experience-Report etc. Thank you! :)
Environment
  • MATLAB R2021b Update 3
  • Windows 10
  • CPU: Intel i7 9700k
  • SSD: SanDisk SDSSDH3 2T00 (Read-Speed: 500MB/s)

 Accepted Answer

I reached out to the Matlab support and they told me the following:
"Thank you for posting this issue. We apologize for any inconvenience this may cause in your project, but this performance issue is a known issue which will be solved with our new MAT-File format which we currently develop for future MATLAB releases.
The background information for the performance issue is the fact that timetables provide functionality that raw matrices don't and they're the pattern adopted by Predictive Maintenance. The tradeoff for that functionality is that constructing them when loading isn't going to be as fast. Extra time comes from constructing timetables because we serialized timetables to MAT files as structs.
Please check the new MATLAB release notes to get more details about the new MAT file improvements if they are available in the future releases."
So right now, there is no way to increase the speed of loading timetables.

More Answers (1)

v7.3 files are considerably slower for composite data types.
v7.3 files are based on HDF5, which does not really have a good way to define a composite type, but has the ability to say "okay, for the value of this element, see this internal variable"... through multiple levels.

1 Comment

I tried to save the data with v7 files and there is indeed an improvement over the v7.3 files. Since v7 files can't store large amounts of data (I received errors trying to save them which makes sense) I needed to reduce the length_run_in_ms from 1.000.000 to 100.000. As shown below, the main problem I described however remains. Reading timetables is significantly slower than numeric data. This just doesn't makes sense to me.
% Compare listing all variable names
tic
signal_names = who("-file", "timetables.73.mat");
toc % Elapsed time is 2.671444 seconds.
tic
signal_names = who("-file", "timetables.mat");
toc % Elapsed time is 1.177656 seconds.
tic
signal_names = who("-file", "numeric.73.mat");
toc % Elapsed time is 0.137482 seconds.
tic
signal_names = who("-file", "numeric.mat");
toc % Elapsed time is 0.053079 seconds.
Since loading all variables at once is for timetables equally fast as loading just the variable names I think MATLAB loads all variables and then gathers their names. For numeric data there seems to be a faster method.
% Compare listing all variable names
tic
signals = load("timetables.73.mat");
toc % Elapsed time is 2.346205 seconds.
tic
signals = load("timetables.mat");
toc % Elapsed time is 1.199220 seconds.
tic
signals = load("numeric.73.mat");
toc % Elapsed time is 0.926132 seconds.
tic
signals = load("numeric.mat");
toc % Elapsed time is 0.815196 seconds.

Sign in to comment.

Categories

Find more on Programming in Help Center and File Exchange

Products

Release

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!