Selecting data from large array

I have a large array (4x700,000) representing minute data of irradiance for over a year. How do I select a days worth of the total to perform some calculations on the irradiance (how it heats up a flat plate panel). I want to call a single day, do the calculations and then save the results to an xls file. Then select the next days worth of data and using the last values calculated in the previous loop start the new loop calculations. I then want to save that days' calculations to an xls file and repeat that for my entire data set. I have constants needed for the calculations as variables, but I can't call the entire data set which is (4 x 700,000) each minute without running out of memory so at the end of each day I need to save the results and then clear the variables for that day except the very last numbers calculated so that I have free memory to perform the next days' calculations. Is there a good way of doing this without calling xlswrite for each iteration (which is every minute) as I'm wanting a daily output. I essentially want to end up with over 365 files representing each day which has minutely data of the calculations I have performed.

Below is part of my code:

%%Get irradiance data
% padded_file = 'C:\Users\User\Google Drive\Jamie help\WORKING_VERSION\irr_data\padded_irr_data_1year.mat';
% if exist(padded_file, 'file')
%   saved_variable = load(padded_file, 'times', 'direct_irr', 'diffuse_irr');
%   times = saved_variable.times;
%   direct_irr = saved_variable.direct_irr;
%   diffuse_irr = saved_variable.diffuse_irr;
% else
  duration_days = floor(end_date - start_date); %number of days running for
  [times, direct_irr, diffuse_irr] = get_irr_data(start_date); %get irr for first day
  for d=1:duration_days %get irr for any extra days after the first day
    this_date = start_date + d;
%     disp(datestr(this_date))
    [tim, dir, dif] = get_irr_data(this_date); %get irr data for this day
    times = [times tim]; %append to existing irr data
    direct_irr = [direct_irr dir]; %append to existing irr data
    diffuse_irr = [diffuse_irr dif]; %append to existing irr data
  end
%   save(padded_file, 'times', 'direct_irr', 'diffuse_irr');
% end

TankTemp = zeros( length(times), constants.NumberSlices );
TankTemp(:,:) = initial_tank_temp;
dt = 1;
TempCoil = zeros(size(TankTemp,1),1); %intitialise the TempCoil vector to have the time dimension
TempCoil(1) = constants.TempCoilInitial;
TempCoil_out = constants.TempCoilInitial;

[dpb]

Are the observations complete for each day in the file?

Irrespective of above, to speed things up significantly use something other than spreadsheet for the output file format.

[Bob Thompson]

dpb has a point about spreadsheet work. If you can save the file as a .txt or .csv then you could read the data using fgetl() which will read the file one line at a time. This normally isn't the best way to do things, but since you can't read the entire dataset at once this would be my suggestion. All you need to do is put fgetl inside a couple of loops, one to examine the data from each day, and one to examine the full year, and make sure you populate an array with the data as needed. fgetl will overwrite itself each time it is used with the same variable name, and you could easily use a clear(dataarray) command at the end of the day loop to remove the stored data from the memory. (or you could just let it run the loop again and be overwritten by the new day loop)

[dpb]

[James' ANSWER moved to comment...dpb]

Thank you for your suggestions, I am going to change the the format of the xls file to a csv. I can also split the data up into "day chunks" and name each day sequentially. The solar observations are complete for the entire data set.

From what you're saying it seems that I need something like:

for  D = 1:365
  filename = 'solarD.csv';
  S =   csvread('solarD.csv',1,1, ;
  {calculations bit here}
  clear(S)
end

Hmmmmm, not sure I'm right! When I clear the data array S at the end of each day I need to keep the very last minute from the previous day 23:59 to start the next day with. Will using Clear S stop me from doing that (sorry not close to Matlab for another day)

[dpb]

What form is the input data? I see commented-out stuff that reads a .mat file; there's memmapfile that may be of significant help if so.

Depending upon what your irradiance calculation is; it may make great difference in how you could write the code if all daily observations are present; that implies a constant number of elements/day which can have great ramifications in being able to use inbuilt Matlab functions' penchant for operating by column if, for example, you could read a week at a time and reshape() and compute mean() all at once.

Just not enough details provided to know precisely what would be most advantageous.

If you do build a sequence of daily files, use a format for the sequential number such as num2str(dayNo,'%03d') or the files will sort alphabetically rather than numerically which will be a real pit(proverbial)a(ppendage) later on when comes to use them.

Answer 1

[James Williams]

Thank you for your suggestions, I am going to change the the format of the xls file to a csv. I can also split the data up into "day chunks" and name each day sequentially. The solar observations are complete for the entire data set.

From what you're saying it seems that I need something like:

for D = 1:365

filename = 'solarD.csv'; S = csvread('solarD.csv',1,1, ;

{calculations bit here}

clear(S) end

Hmmmmm, not sure I'm right! When I clear the data array S at the end of each day I need to keep the very last minute from the previous day 23:59 to start the next day with. Will using Clear S stop me from doing that (sorry not close to Matlab for another day)

[Bob Thompson]

You can always save the last bit of information as a separate variable before deleting 'S'.