Need help with "Out of memory" error

Hi everyone, I'm working with Matlab and InfluxDB in need to make them communicate and work with files containing millions of rows.

When I have to import the csv file with readtable I get no memory problem, but when I get to work with influxdb through matlab I get that a lot. I managed to resolve the problem when writing the data in the database just segmenting the timetable, but when I use the commands that helps me extract the data from the database (around 5 millions of rows) i get that error and I can't find a way around it.

Can someone please help me? Thanks in advance.

Answer 1

[Sam Chak]

5 millions of rows...

Even Excel cannot handle more than 1,048,576 rows. What is the size of your data array?

N = 5e6;
n = round(sqrt(N));
A = rand(n)

A = 2236×2236

0.2871 0.2892 0.2362 0.8154 0.6719 0.9486 0.0286 0.0167 0.0037 0.9809 0.2334 0.3200 0.6854 0.6888 0.1695 0.9808 0.2442 0.7251 0.9363 0.5723 0.9718 0.5492 0.7839 0.9613 0.7380 0.8300 0.1917 0.6323 0.4454 0.1851 0.3742 0.5974 0.1385 0.3654 0.7016 0.4632 0.2757 0.1444 0.1932 0.9558 0.1683 0.4811 0.3795 0.9344 0.0477 0.0531 0.9365 0.8929 0.8917 0.5186 0.2335 0.4736 0.3354 0.0314 0.2693 0.9257 0.4876 0.6192 0.8157 0.0835 0.9680 0.8536 0.5761 0.4397 0.7299 0.4963 0.4466 0.7446 0.1443 0.3364 0.2759 0.5051 0.8596 0.1811 0.9695 0.0002 0.8675 0.0702 0.0458 0.7376 0.0446 0.7774 0.4039 0.0414 0.3167 0.7822 0.3230 0.2219 0.2134 0.8739 0.0221 0.9181 0.7088 0.0396 0.4419 0.8038 0.9004 0.5114 0.8154 0.0024 0.5422 0.6169 0.3261 0.2753 0.1199 0.0807 0.5046 0.4376 0.1845 0.8173 0.7529 0.4920 0.7203 0.8673 0.4909 0.5040 0.0431 0.2915 0.2074 0.6698 0.2228 0.5988 0.0644 0.4645 0.8457 0.3008 0.7301 0.7456 0.3475 0.8645 0.0832 0.5660 0.6291 0.3286 0.5295 0.7372 0.3966 0.8460 0.6487 0.0219 0.9833 0.0456 0.9380 0.3492 0.3861 0.3639 0.9372 0.4164 0.7274 0.6656 0.3047 0.7029 0.6693 0.6465 0.2201 0.7117 0.0093 0.2530 0.9470 0.2857 0.5328 0.3144 0.2076 0.8893 0.1145 0.7776 0.9828 0.7616 0.6983 0.9311 0.2531 0.9205 0.4533 0.5781 0.7889 0.1680 0.8151 0.9566 0.5768 0.7956 0.1690 0.4041 0.2299 0.0682 0.1962 0.4454 0.7184 0.8617 0.5937 0.0149 0.5761 0.2546 0.8043 0.2645 0.7105 0.1647 0.2057 0.4319 0.6486 0.3131 0.0807 0.6536 0.2650 0.6225 0.6905 0.0867 0.1694 0.8908 0.2788 0.9686 0.5583 0.2647 0.6359 0.0701 0.1135 0.1764 0.6828 0.2149 0.9057 0.2459 0.4584 0.2944 0.1062 0.8637 0.2237 0.4451 0.7978 0.6307 0.7105 0.6784 0.0101 0.0640 0.7792 0.0225 0.7677 0.6073 0.2113 0.8333 0.0855 0.1213 0.5186 0.2191 0.3643 0.3197 0.7782 0.6828 0.2533 0.6939 0.0983 0.1220 0.0086 0.6968 0.3239 0.0517 0.4423 0.6635 0.5764 0.5497 0.8355 0.9435 0.8975 0.6853 0.8019 0.0649 0.0176 0.7949 0.8224 0.2203 0.2157 0.3511 0.9187 0.2796 0.7165 0.5693 0.8563 0.6711 0.7393 0.8317 0.8806 0.9828 0.9553 0.1042 0.5307 0.3902 0.9207 0.3834 0.4921 0.5256 0.6935 0.4608 0.2148 0.2966 0.0602 0.5499 0.5002 0.3270 0.8936 0.9960 0.0837 0.0307

B = ones(N, 1)

B = 5000000×1

1 1 1 1 1 1 1 1 1 1

whos

  Name              Size                 Bytes  Class     Attributes

  A              2236x2236            39997568  double              
  B           5000000x1               40000000  double              
  N                 1x1                      8  double              
  cmdout            1x33                    66  char                
  n                 1x1                      8  double              

[Sara Bruschi]

I have a timetable that's 5188521x11, what's strange is that I can import the csv with readtable and work with it without having the error, but when I get them from the database it shows..

Name Size Bytes Class Attributes

T 5188521x11 456594784 timetable

[Sam Chak]

Hmm... 🤨 That's about 456 MB.

Please check if you can increase the array size limit to 100% of RAM.

What does InfluxDB do through MATLAB?

[Sara Bruschi]

I alredy checked that and it was already set as in your photo.

For let matlab and influxdb communicate I'm using a user code found on github: https://github.com/wardgssens/influxdb-matlab/tree/v2-compatibility.

After writing my timetable to influx via the write command I'm trying to call them back with the query command but I get this error if I don't limit the rows.

It's strange also because if I limit the query to 999'999 rows it works fine, but limiting it to 1'000'000 rows I get the same error..