Store all iteration loop outputs in a file

Hi, I am writing a program with a loop where the output from every iteration is a matrix with the same size (the size of the matrix is very large). However, I want to save the output from each iteration in a file (and not to create another variable (matrix or a vector)), as the number of iterations is more than 100 000 (needed for Gibbs sampling). I need to save the output from each iteration in a file (ex. .mat or txt) as storing it in a new variable might consume more RAM than my computer has available. I need to read the results from the file and do further calculation. I was recommended to use 'save' but I don't know how to save the output from all iterations as the name of the matrix (b) is the same and it is overwritten after each iteration, if I use the '-append'.

Simple example of the program is:

a=[1 1 1; 1 1 1; 1 1 1];

b=[];

for i=1:6

    b=a.*i
    save('test.mat','b', '-append')
end
The output should be:
[1 1 1
1 1 1
1 1 1 
2 2 2
2 2 2
2 2 2
3 3 3
3 3 3
3 3 3
4 4 4
4 4 4
4 4 4
5 5 5
5 5 5
5 5 5
6 6 6
6 6 6
6 6 6]

Answer 1

[Stephen23]

If the file is very large then you could use memmapfile:

https://www.mathworks.com/help/matlab/import_export/overview-of-memory-mapping.html

Or one of the other large file accessing tools:

https://www.mathworks.com/help/matlab/large-files-and-big-data.html

Or you can simply create a new file on each iteration:

 save(sprintf('name%04d.mat',k),'X')

Note: the -append option should really be named the -replace option, as the save documentation makes clear: " save(filename,variables,'-append') adds new variables to an existing file. If a variable already exists in a MAT-file, then save overwrites it. The variables argument is optional."

[StudentPoliTo]

My problem is very close to this. I used the suggested solution to create a new file on each iteration:

 save(sprintf('name%04d.mat',k),'X')

But I don't get it in the workspace.

My question is how can I create a matrix with all the results from each iteration, in the workspace from this files created in the folder ?

I tried with 'importdata' but i didn't understand how it works in loops.

[Stephen23]

@StudentPoliTo: saving data does not put it into the workspace! Saving data stores data in files on your harddrive.

The reverse is called importing: this is when you take data from your harddrive (e.g. files) and import it into the MATLAB workspace to work with it.

"My question is how can I create a matrix with all the results from each iteration, in the workspace from this files created in the folder ?"

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

https://www.mathworks.com/matlabcentral/answers/57446-faq-how-can-i-process-a-sequence-of-files

and any of the thousands of other threads on this forum that discuss how to import files in a loop.

[StudentPoliTo]

Thank you very much!!!

[joshua Abam]

Hi

In follow up of your answer is it possible to apply same to iterations from Genetic algorithm, AS I am currently having challenges of saving the iterations from GA to a single file or each iteration to a seperate file

better still use this example to illustrate it and where best can I possible attached that to a GA code.

function f = gaintobj(x)

f = rastriginsfcn([x(1)-6 x(2)-13]);

f = f + rastriginsfcn([x(3)-3*pi x(4)-5*pi]);

lb = [1 1 -30 -30];

ub = [20 20 70 70];

%%

% Set the integer variables and number of variables.

IntCon = [1 2];

nvar = 4;

%%

% Set options to call the custom output function, and to initially have

% little crossover.

options = optimoptions('ga','OutputFcn',@gaoutputfunround,'CrossoverFraction',0.2);

%%

% For reproducibility, set the random number generator.

rng(10)

%

% Set the objective function and call the solver.

fun = @gaintobj;

[x,fval] = ga(fun,nvar,[],[],[],[],lb,ub,[],IntCon,options)

Thanks

Kind regards

Joshua