Clear Filters
Clear Filters

How to organize a datastore with multiple devices for multiple days?

4 views (last 30 days)
So, I have a data from 75 smart meters for range of 17 days.
Each day all 75 smart meters did some measurements.
How it is better to organize the data? I really like the idea of datastore() in Matlab. But don't know how to do it automatically for so many variables(tables)
If I create a timetable for each day of each smart meter, it would require me to create 75*17 =1275 separate tables.
And also I would like to analyze each day (with all smart meter measurements combined) so it will make the total number 1275 +17 = 1292.
Can you suggest a good option of organizing such big amount of data?
I am analyzing it for load forecasting using Machine Learning.
Thank you very much in advance!
  1 Comment
dpb
dpb on 22 May 2022
That's really not much data at all in terms of observations -- unless there are gigabytes of data which it wouldn't seem there could be but a few measurements from each device.
I'd just collect the whole dataset in memory in a timetable and use retime and similar functions to do operations over the data. memapfile is just going to get in the way here.

Sign in to comment.

Accepted Answer

Magsud Hasanov
Magsud Hasanov on 14 Jun 2022
I just used array with all paths to all files and made datastore(), then did readall()

More Answers (1)

Seth Furman
Seth Furman on 30 May 2022
Provided your data are already stored in files on disk in a common format, you can create a tall timetable and perform operations on it as if it were a large timetable.
  1 Comment
Magsud Hasanov
Magsud Hasanov on 13 Jun 2022
Hello, after making path into all files, I just used datastore() and then readall(). It created the big dataset. The amount of data was enough small to use readall(). Thank you very much for your time!

Sign in to comment.

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!