Read Data and Store Variables for Hundreds of Files...

Hi all,

I have a folder with hundreds of files. I would like to read each file individually and store the results so that, in the end, I can average all of my results to produce a matrix I can visualize.

I tried to preallocate and fill a rather large matrix (20 x 600 x 300 x 4 x number of files in folder) with my output to then average along the last dimension, but this exceeds the preferred array limits set forth by MATLAB. So, there is insufficient space on my program and computer to store an array of this size.

In essence, I would like to end up with a 20 x 600 x 300 matrix that is the average of all of the timesteps (4) included in my hundreds of files. The problem is, I don't know how best to work around the space constraints. Any help would be greatly appreciated.

A bit more on the variable I am interested in extracting from each file: the variable ('temp") has four dimensions, representing the number of levels (20), longitude (600), latitude (300), and timesteps in a day (4). In other words, I am trying to read the temperature values at all locations over all 20 levels and all four timesteps in a day (one file == one day). I then need to average all of these results along the 4th dimension (time) to produce a final 20 x 600 x 300 average of temperature over all of my files. Here is what I have so far:

Folder = 'C:\Users\my\Desktop\Folder'
FileList = dir(fullfile(Folder, '*.nc4'))
temp = zeros(21,576,361,4)
size(temp)
for iFile = 1:numel(FileList)
    iFile
    filename = fullfile(FileList(iFile).folder, FileList(iFile).name)
    for i = 1:20
        i;
        temp(i,:,:,:) = ncread(filename, 'temp', [1, 1, i, 1], [inf, inf, 1, inf]);
    end
end

Answer 1

[Walter Roberson]

Folder = 'C:\Users\my\Desktop\Folder';
FileList = dir(fullfile(Folder, '*.nc4'));
filenames = fullfile({FileList.folder}, {FileList.name});
total = 0;
nfile = numel(filenames);
for iFile = 1:nfile
    filename = filenames{iFile};
    thisdata = ncread(filename, 'temp');
    if iFile == 1
       total = thisdata;
    else
        total = total + thisdata;
    end
end

%total is now a per-file total. Need mean per timestamp
mean_data = sum(total,3) ./ (nfile*size(total,3));
mean_data = permute(mean_data, [3 1 2]);    %user wants level first

[Michelle De Luna]

Walter, as always, thank you for your support! Your answer helped me approach my question with a different perspective. Your answer of summing the matrices and then finding the mean was correct and to-the-point! Again, thank you.

[Walter Roberson]

mean_data = sum(total,3) ./ (nfile*size(total,3));

should probably be 4 instead of 3 in both places.

Also if ncread is returning integer datatype then double(thisdata) or else you get integer overflow.