Error using zeros and out of memory

Hi everyone, a friend of mine helped me write this code. The code works but, if I increase the values on lines 11, 12 and 13, I get this error message:

Error using zeros Requested 624x103x103x300 (14.8GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

Can you help me solve this problem? The processing of all the code really takes a very long time, I think it can certainly be written better by reducing processing time but I do not know how to do it.

Thank you.

[jonas]

The error message is very clear, I don't know what else to say to be honest. You try create an array with 2 billion elements, which is too much to store in memory.

[Stephen23]

"Can you help me solve this problem?"

You have three main choices:

1. buy more memory, or
2. process less data, or
3. change your algorithm.

Which do you want to do?

[Marco Tavecchia]

3 - Change my algorithm

[Rik]

Have you tried using the profiler to find out where the bottleneck is for your function? Your code is quite lengthy and since the comments are in Italian(?) I don't understand them enough to understand your algorithm and propose a solution.

[Marco Tavecchia]

Thanks.

Yes, the code spend much time in this point:

line 4 and line 20 - xlsread

[Rik]

@Jan, be happy that the code was included as an attachment, otherwise you'd spend quite some time scrolling...

@Marco, those calls to xlsread are not repeated, so those can only be sped up by not storing the data in excel files, but store them in a mat file instead (checking every time if the mat file is newer to make sure no edits to the excel file are missed).

[Marco Tavecchia]

the code reads the data in the two excel files that I have attached. If I understand your suggestion, should I copy the data from the Excel file and paste it into a mat file? If the answer is yes, how can I do this? Thank you

[Marco Tavecchia]

I tried to do this but it doesn't work

[num,txt,raw] = xlsread("49giorni - 5 day.xlsx")

AllData={txt;num}

save(49giorni5day.mat,'AllData')

[Walter Roberson]

save('49giorni5day.mat','AllData')

[Guillaume]

Marco, I don't think you should focus on the way you store your inputs/outputs for now. Yes, using excel is not particularly efficient but it's a minor problem compared to the big one:

Your current processing algorithm (which I've not tried to understand) requires the creation and processing of lots of very large 4D matrices. In the beginning of your code I counted the creation at least 24 matrices of size 624x103x103x300 (If your comment is correct about the sizes) which is going to require around 355 GB of memory alone.

You need a fundamental change of algorithm, not minor edits here and there. You need to come up with an algorithm that doesn't require storing/generating so much data. Or you need to run your algorithm on a supercomputer, standard desktop computers simply won't cut it for that.

[Marco Tavecchia]

@Walter

doing so I get a .mat file 2 rows and 1 column while I would like a file with 5617 rows and 9 columns.

>> AllData={txt;num}

AllData =

2×1 cell array

    {5617×9 cell  }
    {5616×6 double}

[Marco Tavecchia]

@Guillaume

Yes, I have much 4D matrices and the code works very slow.

if you have any suggestions I try to start editing something

[Stephen23]

"if you have any suggestions I try to start editing something"

Guillaume already explained what you need to do:

https://www.mathworks.com/matlabcentral/answers/428691-error-using-zeros-and-out-of-memory#comment_634880

"You need a fundamental change of algorithm, not minor edits here and there"

[Walter Roberson]

 I would like a file with 5617 rows and 9 columns.

However that is the size of your txt by itself. Where would num be put?

[Marco Tavecchia]

I want the code read the values with the same references in the positions that had the data in Excel

[Walter Roberson]

AllData = raw(:,1:9);

[Marco Tavecchia]

Meanwhile, I want to thank everyone for the suggestions and patience shown me.

I'm trying to rewrite the code from the beginning.

The first thing I do is transform the excel file "input_mat2.xlsx" into a matlab file "input_mat2.mat".

I did this way:

>> [num,txt,raw] = xlsread("input_mat2.xlsx")

>> AllData = raw(:,1:3)

>> save('input_mat2.mat','AllData')

If I open the file "input_mat2.mat" all is in the correct position.

Now, returning to the original code, I want to create the variable "input_mat" and read some values contained in the file "input_mat2.may" in this way:

>> input_mat = load('input_mat2.mat')

>> alfa = round(input_mat(6,2))

This unfortunately I can not, I should get as a response the value "104" that corresponds to the value in the sixth row-second column of the file "input_mat2.mat", while I get this error message:

>> alfa = round(input_mat(6,2))

Index in position 1 exceeds array bounds (must not exceed 1).

[Rik]

The load function loads the variables inside the file to a struct, so you still need to parse that:

input_mat = load('input_mat2.mat'); AllData=input_mat.AllData;
alfa = round(AllData(6,2))

[Marco Tavecchia]

Thank you.

I have done as you indicated but it gives me this error:

>> alfa = round(AllData(6,2))

Error using round

First argument must be a numeric, logical, or char array.

[Marco Tavecchia]

So instead I do not give error and I get the value sought (104):

alfa = AllData(6,2);

it's correct?

[Guillaume]

Marco, in your original comment you said: "I'm trying to rewrite the code from the beginning"

Yes, you should do that. However, as I said originally, you should not worry about how you get your data into matlab for now. That's a minor detail at this point. What you need to fundamentally change is what you do with that data once it's loaded. You cannot use it to generate over 350 GB of matrices. Even one matrix of 15 GB is most likely to tax your computer and be slow to process.

Once you have a processing algorithm that doesn't require so much data, then you can focus on speeding up the import, if it's still an issue.

[Marco Tavecchia]

I know but I try to check each line.

[Marco Tavecchia]

I have made the changes mentioned in the comments above.

In the original code, the "percent" variable did not generate an error, but now it gives me this error:

>>Estremo_Superiore_Variazione_Percentuale = AllData(11,2);

>>Estremo_Inferiore_Variazione_Percentuale = AllData(12,2);

>>Passo_Variazione_Percentuale = AllData(13,2);

>>percent=[Estremo_Inferiore_Variazione_Percentuale:Passo_Variazione_Percentuale:Estremo_Superiore_Variazione_Percentuale]

Undefined operator ':' for input arguments of type 'cell'.

[Marco Tavecchia]

The dimensions of variable "percent" is 1x301 (percent=[30% : 0% : 0,1%])

0,0% - 0,1% - 0,2% ...29,8% - 29,9% - 30,0%

[Walter Roberson]

 Estremo_Superiore_Variazione_Percentuale = AllData{11,2};

And similar changes. The raw output of xlsread is a cell array.

[Marco Tavecchia]

I solved this way, so doing I display all 300 values

Estremo_Superiore_Variazione_Percentuale = AllData{11,2}; %  30,0%
Estremo_Inferiore_Variazione_Percentuale = AllData{12,2}; %  0,0%
Passo_Variazione_Percentuale = AllData{13,2}; %  0,1%

percent = [Estremo_Inferiore_Variazione_Percentuale:Passo_Variazione_Percentuale:Estremo_Superiore_Variazione_Percentuale]

Thanks Walter

[Marco Tavecchia]

Well

Now let's start with the problems of the first 4D matrix

>> buytendenza_conf_sum = zeros(gamma,delta,delta,length(percent))
Error using zeros
Size inputs must be numeric.

[Marco Tavecchia]

These are the values ​​of gamma, delta, delta, lenght (percent)

>> gamma,delta,delta,length(percent)
gamma =
  1×1 cell array
    {[624.0000]}
delta =
  1×1 cell array
    {[103.0000]}
delta =
  1×1 cell array
    {[103.0000]}
ans =
   300

[Rik]

It looks like your values are stored in cells, so they are not numeric, hence the error.

[Guillaume]

As Rick says, somehow you've managed to store your gamma,delta, etc. in cell arrays. What's clear though is that their value has not changed and you still requires massive amounts of memory.

This is the last time I'm saying this: You need a fundamental change of algorithm. You need to throw away your current code and start fresh with a completely new way of processing your data. Changing your current code one line at a time is not going to work.

In my opinion, you would be better off starting a new question explaining what your inputs are and what you want to calculate from them and let us come up with an algorithm.

[Marco Tavecchia]

Ok, you convinced me, I'll try to figure out how to ask the question as clearly as possible.

[Stephen23]

@Marco Tavecchia: and please format your comments properly. Formatting the entire comment text with monospaced typeface does not make your comments easy to read and follow. Learn to use that nice toolbar above the textbox: do you notice it has a section for formatting code?

Today I formatted your last few comments for you. In future you can do it yourself.

[Marco Tavecchia]

Ok Stephen, sorry.