How to organize a datastore with multiple devices for multiple days?

So, I have a data from 75 smart meters for range of 17 days.

Each day all 75 smart meters did some measurements.

How it is better to organize the data? I really like the idea of datastore() in Matlab. But don't know how to do it automatically for so many variables(tables)

If I create a timetable for each day of each smart meter, it would require me to create 75*17 =1275 separate tables.

And also I would like to analyze each day (with all smart meter measurements combined) so it will make the total number 1275 +17 = 1292.

Can you suggest a good option of organizing such big amount of data?

I am analyzing it for load forecasting using Machine Learning.

Thank you very much in advance!

[dpb]

That's really not much data at all in terms of observations -- unless there are gigabytes of data which it wouldn't seem there could be but a few measurements from each device.

I'd just collect the whole dataset in memory in a timetable and use retime and similar functions to do operations over the data. memapfile is just going to get in the way here.

Answer 1

[Magsud Hasanov]

I just used array with all paths to all files and made datastore(), then did readall()

Answer 2

[Seth Furman]

Provided your data are already stored in files on disk in a common format, you can create a tall timetable and perform operations on it as if it were a large timetable.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

[Magsud Hasanov]

Hello, after making path into all files, I just used datastore() and then readall(). It created the big dataset. The amount of data was enough small to use readall(). Thank you very much for your time!