Store several output matrices in a single matrix

Hi everybody!

I have 205 .mat files of size 31590x6 and I have a script that takes the averages every 15 minutes so that every file is reduced to a size 34x6. So my main goal after calculating the 15 min intervals is to put all these reduced files into a single matrix that I can use later for other calculations.

The following script does the interval calculations without problems, but when I triy to put all the Oupt matrices of each file in a single matrix, it doesnt work :( because matlab says : ??? Index exceeds matrix dimensions.

All_Data=[];
dataFiles=dir('*.mat');
for d=1:length(dataFiles)
  file=dataFiles(d).name; %gets names of all mat files
  fileData=load(file);
  Data=fileData(d).finalData;

    %NAME DATE, STOCK AND TIMESTAMP

    date=Data(2:end,1);
    timestamp=Data(2:end,2);
    stock=Data(2:end,3);

    %NAME BID VARIABLES
    bid1=cell2mat(Data(2:end,4));

    %NAME ASK VARIABLES
    ask1=cell2mat(Data(2:end,14));

    %NAME BID VOLUME
    bvol1=cell2mat(Data(2:end,24));

    %NAME ASK VOLUME
    avol1=cell2mat(Data(2:end,34));

    %CONVERT TIME STAMP AND DATE INTO NUMERIC VALUE
    dateNum=datenum([date{:}],'dd.mm.yy');
    timeNum=datenum([timestamp{:}],'HH:MM:SS');

    %CALCULATE TIME INTERVALS
    t = dateNum+timeNum;
    t15 = datenum('00:15:00','HH:MM:SS')-datenum('00:00:00','HH:MM:SS');
    ngroups = ceil((max(t)-min(t))/t15); %number of bins 
    findfirsttime =find(timeNum,1,'first');
    firsttimestamp=timeNum(findfirsttime);
    differences=zeros(size(t)); %assign memory
    differences = timeNum(:)-firsttimestamp(:);
    binNumbers=differences*96;
    FifteenMinBins=floor(binNumbers);

    %CALCULATE BID MEANS
    b1=accumarray(1+FifteenMinBins(:),bid1(:),[],@mean);

    %CALCULATE ASK MEANS
    a1=accumarray(1+FifteenMinBins(:),ask1(:),[],@mean);

    %CALCULATE BID VOL MEANS
    bv1=accumarray(1+FifteenMinBins(:),bvol1(:),[],@mean);

    %CALCULATE ASK VOL MEANS
    av1=accumarray(1+FifteenMinBins(:),avol1(:),[],@mean);

    %CALCULATE TIME INTERVAL FOR TIME
    timeinterval=accumarray(1+FifteenMinBins(:),timeNum(:),[],@mean);

    %GET FIRST 34 NUMERIC VALUES OF DATE
    getdatenum=dateNum(1:34);

    %COLLECT OUTPUT IN MATRIX

    %Outtput=[getdatenum timeinterval b1 a1 bv1 av1];
    %save('Output(d).mat',Output)

    %STORE OUTPUT IN A MATRIX
    Output=[getdatenum timeinterval b1 a1 bv1 av1];

    %for i=1:length(dataFiles)
    Output=[getdatenum timeinterval b1 a1 bv1 av1];%i try w/4 var
    All_Data=[All_Data;Output];

end

The error I get is the following :??? Index exceeds matrix dimensions. And when i check the All_Data matrix it just contains one matrix of size 34x6, but i cant tell if it is the first matrix or the last one when the scripts runs the part of calculating the time intervals for all those 205 files.

So in the end, the last part of the code is trying to get a matrix of size 6970x6; composed of the output matrices of all the reduced files.

Do you see something wrong in the code? I don´t understand what the problem is and I have been trying it so many times that i don´t k now what else to do.

I would apprecciate any help.

Thank you and have a nice day!

Lourdes

[Jan]

"it doesnt work" is not a precise description of the problem. Please post the error message, because it usually contains important hints for the needed solution.

[Jan]

"it doesnt work" is not a precise description of the problem. Please post the error message, because it usually contains important hints for the needed solution.

The 11 out-commented lines at the end of the loop are confusing. Please edit your question and remove everything, which is not related to your problem.

[Lu]

Dear Jan,

Thank you for the reply! I edited my question and erased the comments. I hope it is more clear now. Do you have any ideas of why the Ouput matrix of each file is not appending one below the other one?

Have a nice day,

Lourdes

[Oleg Komarov]

This is a duplicate to: http://www.mathworks.com/matlabcentral/answers/7548-problem-in-appending-matrices-in-vertical-form

Please don't post duplicates. Also, you didn't correct for my observations in the previous posts and and "end" is missing here and redundant "Output" line is added here.

Note that my comments on the other post is the same as Fangjun's.

[Lu]

Thank you for your helpful suggestions

Answer 1

[Fangjun Jiang]

You have a nested for-loop. You don't need the inner for-loop for this purpose.

You have the variable "All_Data" set to be empty at the beginning. You go through a loop for 205 files. Every loop you calculate the variable "Output" and then you append it to "All_Data". Your coding flow is sound.

Your code should work if you remove your inner for-loop. Just leave the line "All_Data=[All_Data;Output];" there.

A couple of advice: Don't use variable "date". It is a built-in function. Don't use confusing variable names like "All_Data" and then "All_data". In this case, "All_data" is never used. I believe it is a mistake. Remove it.

[Lu]

Dear Fangjun Jiang,

Thank you for your answer. I followed your tips and erased the All_data=[] and also erased the little loop and just left

All:Data=[All_Data;Output], however matlab says :

??? Index exceeds matrix dimensions.

Do you have an idea of why this might be happening?

Thank you so much!

Lourdes

[Fangjun Jiang]

Can you re-post your current code? And also, paste the complete error message, which usually indicates which line the error occurred.

[Lu]

Dear Fangjun Jiang,

I re-posted my current code and pasted the only error it shows, which is : ??? Index exceeds matrix dimensions.

That´s all it says. Do you see something wrong in the code that wont let the matrices append into a single matrix?

Thank you so much :)

[Fangjun Jiang]

I don't see any obvious problem. Can you step through your code to see which line the error first occurred? Or is it possible that you construct a simple set of data so others can help to debug?

[Lu]

Dear Fangjung Jiang,

I don´t really know much matlab, so i don´t know how to step in my code and see in which line the error occurres. How about this:

g=rand(34,6); generates random numbers of the same size as my Output matrix

e=rand(34,6); does the same but let´s say it is the output matrix from a different file

f=rand(34,6); output matrix from a third file

Is this random data enough? Probably the time intervals calculation should be ignored because the script runs that correctly. The problem comes when appending the Output matrices for all files in one All_Data matrix.

Thank you so much :)

[Lu]

Or I can send you just 2 files and then you could run the entire script? It is up to you, but thank you so much for taking time in helping me out :)

[Fangjun Jiang]

I can generate these rand() numbers. But some columns are data, or time, or strings so it takes time.

In your M-Editor, click the column just right of the line number, you'll see a red dot come on and off. This is the indication of a breakpoint. Go to menu "Debug", you'll see many of the debugging menu. Press F5 to start running your code. It will pause at your breakpoint. Press F10 to step through your code line by line. Observe messages or variables in the Matlab command window. This is how do you debug your code. You need to learn this to effectively develop your code.

[Lu]

All the output stored in the matrix Output are doubles because even dates and times are in numeric format, so that´s why I thought using rand() to generate a data set.

I followed your instructions in stepping-in my code and I put the red dot in the line of: All_Data=[All_Data;Output];

matlab says that : ??? Undefined function or method 'INTERVALS' for input arguments of type

'char'.

I dont know why because the script is not a function and I dont use the word INTERVALS for anything, except for the file name and some comments that I write using %creating intervals.

[Fangjun Jiang]

Put the breakpoint on your fist line and step through. Try to catch which line does the error first occurred. The error sounds like that you are trying to use a char variable as an interval to create a matrix.

[Lu]

I put the breakpoint on the first line but the command of step in appears in gray. However, I think the problem is the way I´m trying to load all the files in the beginning. For some reason this part is not working, because when I just type this:

All_Data=[];

dataFiles=dir('*.mat');

for d=1:length(dataFiles)

file=dataFile(d).name;

fileData=load(file);

Data=fileData(d).finalData;

All_Data=[All_Data;Data];

end

matlab throws this error: ??? Undefined variable "dataFile" or class "dataFile".

So I guess it is not loading the files.

Is there another way I can try loading or opening the files ? Or what exactly is wrong in the way I load the files?

when I load any .mat file into workspace individually, it loads the data set (finalData) right away, but if I use my for loop, it loads it as an structure and then to access it the data set i have to type

finalData=fileData(d).finalData.

Do you have any other suggestions on how can I open or load the files in a loop for my script to run correctly?

Thank you so much :)

[Fangjun Jiang]

The breakpoint is grey because your file is modified but not saved yet.

The problem you had is because your create the variable "dataFiles" but use it in "dataFile(d)". This is just a typo but it indicates that you have a lot of debugging to do. You can't expect to write the code and then the code will run without a glitch, can you? Following the debugging method, you'll be able to eliminate the problem one by one. You will learn a lot from this process.

With or without structure, it is just the difference of the syntax using the function load. Don't pay attention to it. That's not the problem. You already know how to handle it.

[Oleg Komarov]

The variable is "dataFiles" and not "dataFile".

[Lu]

Hi!

Thank you for your time and suggestions.