Fastest Way to Keep the last N Outputs of a Loop in a Matrix

Hey everyone, I have some code with a for loop where I would like to store the last N outcomes and do calculations intermittently with the last N evaluations within the loop. I am storing each evaluation result in a column of the matrix mydata. When the number of steps passes N, I remove the first column of data (as it is the oldest) and add my result at N+1th step to the final column as following:

mydata(:,1)=[];
mydata(:,N)=newdata;

While this works fine, it is very time-consuming. Is there a faster way to remove the first column of data and add the new data to the last column of my matrix? Note that I cannot keep all the data as it would end up being a very large matrix.

[dpb]

The classic solution is to use circular buffer and just keep track of the indices instead of moving any data.

You might try to see how well MATLAB does in comparison with

mydata(:,N)=[mydata(:,2:N-1) newdata};

as a sutbstitute intermediary step to making the complete changeover.

[Jan]

@Hakan Caldag: Is it a FOR or WHILE loop? If you know the number of iterations in advance, it is trivial to store the results of the last N iterations only.

[Hakan Caldag]

@dpb I tried your solution but unfortunately there is not much improvement.

@Jan It is a for loop. The for loop lasts, let's say 2e4 iterations and I would like to have the last 3000 results every time.

The circular buffer is a great idea, will try implementing that.

[dpb]

Not surprising but thought just outside chance the optimizer might be able to make it not too bad...not much lost to try and I had to go at the time...

Answer 1

[Jan]

If you do not know the number of iterations in advance, a circulare buffer is more efficient than shrinking an expanding an array repeatedly:

N       = 10;
Buffer  = nan(N, 5);
iBuffer = 0;
while rand > 0.05
    data = rand(1, 5);
    
    iBuffer = mod(iBuffer, N) + 1;  % Wrap around cursor
    Buffer(iBuffer, :) = data;
end
LastData = Buffer([iBuffer+1:N, 1:iBuffer], :);

[Hakan Caldag]

Using a circular buffer works, thank you so much!

[Jan]

If it is a FOR loop, the implementation is easier and more efficient:

N      = 3000;
Buffer = nan(N, 5);
offset = 2e4 - 3000;
for k = 1:2e4
    data = rand(1, 5);
    
    if k > offset
        Buffer(k - offset, :) = data;
    end
end

[Hakan Caldag]

What I wanted was to have the latest N datapoints at each step (whenever k is greater than N). This code seems to work only for when k is larger than offset. In any case, using the circular buffer worked for me already, I just wanted to note this down for anyone that may come across this page in the future.

[Jan]

The circular buffer is filled in each iteration, but only the last N iterations matter. If the number of iterations is known in advance, it is cheaper to fill the buffer only in the last N iterations. The offset is adjusted dynamically to match the number of iterations and the wanted length of the buffer.

The circular buffer is working also, but it wastes time for overwriting memory.