How to use datetime to plot a range of dates

I am looking to plot data from a range of dates, the code used to plot the data begins after the first for loop, which covers everything until the end of the code. I can use the datetime function to print out all of the days but I am not sure how to implement it in my code to actually plot the entire range of days, which in this case goes from 3-15-17 to 4-4-17. The gauge data is in excel files and the satellite data is in text files which both have the same mm-dd-yr name format. I have an example plot at the bottom of the code for reference. I can clarify anything if needed.

 %This code will plot data from a range of dates

 %Convert variable year from integer to string
yr=num2str(17);
dy=num2str(15);
mn=num2str(3);

 yr2 = num2str(17);
dy2 = num2str(4);
mn2 = num2str(4);

 %Create a MMDDYY string 
date_str=[mn '-' dy '-' yr];
date_str2=[mn2 '-' dy2 '-' yr2];

 for day = datetime(17, 3, 15):datetime(17, 4, 4)
day = datestr(day, 'mm-dd-yy');

 %Gauge data location.
gauge_data='C:\Users\bbush\Desktop\Matlab Code\Excel Rain Gauge Data\Daily Rain Accumulation\';

 %GOES data location
GOES_loc='C:\Users\bbush\Desktop\Matlab Code\Satellite Rainfall Data\';

 %Find GOES data
GOES_fn=[GOES_loc,date_str,'.txt'];

 %Find gauge data
gauge_fn=[gauge_data,date_str,'.xlsx'];

 %Read in gauge data
 [num,raw] = xlsread(gauge_fn); %Reads all data from excel file where num will eventually just be the RNFL totals
latG = num(:,1);
lonG = num(:,2);  
RNFLG = num(:,3);
Gauge = cat(2,latG,lonG);

 % read in Satellite data
fid = fopen(GOES_fn);
data = textscan(fid,'%f%f%f%f','HeaderLines',0,'CollectOutput',1);
data = data{:};
fid = fclose(fid);
% retrieve columns
data(:,:);
latS = data(:,1);
lonS = data(:,2);
RNFLS = data(:,3);
Satellite = cat(2,latS,lonS);

 gLon =-83.275:0.025:-82.85;
gLat = 35.3:0.025:35.8;
Z = griddata(lonS,latS,RNFLS,gLon,gLat');
z = log2(Z);

 %Plotting
figure
clf;
load rainmap 
cmap = colormap(rainmap);
rainmap = rainmap(2:2:end,:); %Gets ten RGB triplets 
levels = 2.^(-3:7); %11 levels

 %Plot satellite contours
contourf(gLon,gLat,z,'LineStyle','None') 
hold on
caxis([-3 7])

 %Plot the Gauge data on top
%Plot No RNFL
idx = find(RNFLG==0);
plot(lonG(idx),latG(idx),'o','Color','k','LineWidth',0.5,'MarkerFaceColor','w','MarkerSize',6); %Empty circle
%Plot Small amount of RNFL
idx = find(RNFLG>0 & RNFLG<0.1250);
plot(lonG(idx),latG(idx),'o','Color','k','MarkerFaceColor','k','MarkerSize',6);  %Black circle

 for r=1:10 %Ten RGB triplets for coloring rainfall totals
    idx = find(RNFLG>levels(r) & RNFLG<levels(r+1)); %Find rainfall totals between level ranges
    clr = rainmap(r,:);
    plot(lonG(idx),latG(idx),'o','Color','k','MarkerFaceColor',clr,'MarkerSize',6,'LineWidth',1);
end

 idx = find(RNFLG>levels(r+1)); %Find rainfall totals greater than the level range
clr = rainmap(r,:);
plot(lonG(idx),latG(idx),'o','Color',clr,'MarkerFaceColor',clr,'MarkerSize',6,'LineWidth',3);

 hold off

 cbh = colorbar ; %Create Colorbar
cbh.Ticks = linspace(-3, 7, 11) ; %Create 11 ticks from -3 to 7
cbh.TickLabels = num2cell(levels);  %Replace the labels of the ticks with the values in level

 xlabel('Lon')
ylabel('Lat')
title([date_str,' RNFL Comparison'])

 c = load('coast');
hold on;
u = load('us_state_map.mat');

 plot(c.long,c.lat,'k')
plot(u.state_lon,u.state_lat) 

 xlim([-83.27 -82.88])
ylim([35.33 35.77])

 end

Ive also attached three rain gauge files and three satellite files if anyone is interested in trying themselves.

[dpb]

I'd suggest converting from xlsread, textscan and the resulting cell arrays to readtable and the resulting table objects that have much to commend for ease in accessing.

As for the date selection, read the dates from the spreadsheet as datetimes and then you can simply do indexing operations on that variable to select the range wanted.

To plot a range, simply create a logical vector of those elements wanted and call plot with the indexed array...

ix=(t>=t1 & t<=t2);         % logical vector of those between t1, t2
scatter(long(ix),lat(ix),c) % plot selected w/ color vector

If data files aren't humongous and you can, attach and somebody may have time to look at in more detail...

Read up on the details of using datetime for calculations with dates and times for more background on the above comparisons and also search for "logical addressing" for more on how that works.

[dpb]

Brandon Bush Answered

"Thanks for your answer, I appreciate you having the patience to look through my mess up there. I will certainly try your method. "

[dpb]

Ah! I see the attached files...on reflection I see that's what you described but not quite what I imagined. :) However, it's not that far off to do what I had in mind I think, let me play for a few minutes or so...

[Brandon Bush]

I found this page...

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

the code here lets me plot the data and import the data all the same but only for one file specified by date_str. I cant get it to work for the range of files that I want. it just puts the data from the one file in the mydata cell array the number of times specified by numfiles.

Answer 1

[dpb]

OK, since this is so much altered from the first Q? that I thought was being asked, I'll just make it a second entirely independent Answer. I think the above is still potentially useful first to show how to manipulate the strings to get datetimes from and also to extend and pick up for the sanity check on matching files.

The attached m-file ran here without all the .mat files I don't have so I presume it will run when given those as inputs. I didn't refactor as much as really should be; I left the existing variables as are to simply be able to leave your code for all the plot stuff.

I did move the loop-invariant stuff (what of it I saw, anyway) out of the loop up to the top.

Overall, there isn't really anything changed except to use dir and loop through the returned file list. I didn't really try to check if the file-specific labelling or anything like that works; without being able to run to completion with the missing files that was more than cared to try to patch to debug.

ADDENDUM

You could simplify the plotting quite a bit by the use of histc or, with some machinations, the new histcounts and/or discretize

The idea is something like

bins=[0 eps levels];   % add zero and <levels(1) bins to levels specified
[n,bin]=histc(G.Accumulation,bins);  % find bins

Then you can assign colors to each point by the bin number and make a single call to scatter with that color vector. The colors vector does have to be augmented to cover the two other bins of zero ('w') and nonzero but less than first level ('k') but that's simple enough. Also, histc adds one RH bin to catch any exceedance of the last edge; if it is certain that is a never-occurring event then you can discard it.

[Brandon Bush]

I removed some duplicate code from the first example and re ordered some of the code. The plot prints out the same. I will first place the code and then put details underneath.

D1=datetime('3/15/2017'); D2=datetime('3/16/2017'); % set the desired date range

 gLon= -83.275:0.025:-82.85;
 gLat=  35.300:0.025: 35.80;
 load rainmap 
 cmap = colormap(rainmap);
 rainmap = rainmap(2:2:end,:);     %Gets ten RGB triplets 
 levels = 2.^(-3:7);               %11 levels
 c = load('coast');
 u = load('us_state_map.mat');

 %Gauge data location.
gauge_data='C:\Users\bbush\Desktop\Matlab Code\Excel Rain Gauge Data\Daily Rain Accumulation\';
%gauge_data=cd;    % for dpb
%Find gauge data
gauge_fn=dir(fullfile(gauge_data,'*-*-*.xlsx'));    % return the list of all files
dn1=datetime(extractBefore(cellstr(char(gauge_fn.name)),'.'),'InputFormat','M-d-yy'); % get datetime array
d1=gauge_fn(ismember(dn1,datetime(D1:D2)));                    % save only those between D1, D2

 %GOES data location
GOES_loc='C:\Users\bbush\Desktop\Matlab Code\Satellite Rainfall Data\';
%GOES_loc=cd;      % for dpb
%Find GOES data
GOES_fn=dir(fullfile(GOES_loc,'*-*-*.txt'));
dn2=datetime(extractBefore(cellstr(char(GOES_fn.name)),'.'),'InputFormat','M-d-yy'); % get datetime array
d2=GOES_fn(ismember(dn2,datetime(D1:D2)));                    % save only those between D1, D2

 for i=1:length(gauge_fn)                              % iterate over those desired only
  %Read in gauge data
  g=readtable(gauge_fn(i).name);
  latG = g.Lat;                         % just to keep existing names
  lonG = g.Lon;                         % in end adapt to use directly
  RNFLG = g.Accumulation;               % no need for temporaries
  Gauge = cat(2,latG,lonG);

    % read in Satellite data
    data= readtable(GOES_fn(i).name);
    data.Properties.VariableNames={'Lat','Lon','RNFLS'};
    latS = data.Lat;                      % same thing for temporary variables
    lonS = data.Lon;
    RNFLS = data.RNFLS;
    Satellite = cat(2,latS,lonS);
    Z = griddata(lonS,latS,RNFLS,gLon,gLat');
    z = log2(Z);

    %Plotting
    figure

    %Plot satellite contours
    contourf(gLon,gLat,z,'LineStyle','None') 
    hold on
    caxis([-3 7])

    %Plot the Gauge data on top
    %Plot No RNFL
    idx = find(RNFLG==0);
    plot(lonG(idx),latG(idx),'o','Color','k','LineWidth',0.5,'MarkerFaceColor','w','MarkerSize',6); %Empty circle
    %Plot Small amount of RNFL
    idx = find(RNFLG>0 & RNFLG<0.1250);
    plot(lonG(idx),latG(idx),'o','Color','k','MarkerFaceColor','k','MarkerSize',6);  %Black circle
    for r=1:10 %Ten RGB triplets for coloring rainfall totals
      idx = find(RNFLG>levels(r) & RNFLG<levels(r+1)); %Find rainfall totals between level ranges
      clr = rainmap(r,:);
      plot(lonG(idx),latG(idx),'o','Color','k','MarkerFaceColor',clr,'MarkerSize',6,'LineWidth',1);
    end

    idx = find(RNFLG>levels(r+1)); %Find rainfall totals greater than the level range
    clr = rainmap(r,:);
    plot(lonG(idx),latG(idx),'o','Color',clr,'MarkerFaceColor',clr,'MarkerSize',6,'LineWidth',3);
    hold off

    cbh = colorbar ; %Create Colorbar
    cbh.Ticks = linspace(-3, 7, 11) ; %Create 11 ticks from -3 to 7
    cbh.TickLabels = num2cell(levels);  %Replace the labels of the ticks with the values in level
    xlabel('Lon')
    ylabel('Lat')
    title([date_str,' RNFL Comparison'])
    hold on;

    plot(c.long,c.lat,'k')
    plot(u.state_lon,u.state_lat) 
    xlim([-83.27 -82.88])
    ylim([35.33 35.77])
   end

When running the for loop, the index value i gets set to 2 by length(gauge_fn), which is then used by gauge_fn(i) to pick the file. When the data is loaded into the gauge_fn and GOES_fn variables, the file that is stored in index 2 is 3-11-17 and not 3-15-17 because of the way the files are loaded into matlab. Is there a way that we can have the program search through using just the file name instead of the index?

The gauge data loads in from the file correctly and plots correctly, I will place the plots at the bottom of this message as well as the .mat files so that you can see the colors.

The GOES data in the data variable loads in five columns, two of which are NaN. The data in the Lat and Lon variables are correct, however, the data in the RNFLS variable only comes out to 0. When I add a forth and fifth variable name to data.properties.VariableName the data variable becomes a 3 column table instead of 5, however, the data in RNFLS remains 0 and the plot no longer shows. I did try changing the data.RNFLS variable to match the extra variable names but there was no change to the data.

There are two plots that I have uploaded. One is a blank figure that shows up whenever I run the program and the second is the plot with the data from March 11 and not March 15.

[dpb]

The loop upper index is 2 because you set D1,D2

D1=datetime('3/15/2017'); D2=datetime('3/16/2017');.

such that only files that match are left in the kept directory structure and that is only two days; hence the loop only runs twice.

I don't have anything but the three files attached for testing so I just set those two days to check the logic when my first understanding was that it was that you wanted to only do a subset of all available files.

However, this looks like it was the earlier version, not the last that I attached...please try the code in the attached file instead; it's not the same as the former text that I elided for brevity in the window; it has a couple of fixes including not making that selection; it process all files it finds.

The index to the file in the directory structure is that which is associated with the given dir() order which is in lexical alphabetical order. I don't believe in the updated code there should be any way to have a mixup between dates unless as noted before there's not actually a 1:1 correspondence of there being a matching Excel and text file for every case--I noted there should be error checking for the case where that isn't so but I made the presumption there would be.

The files that you attached read OK here; if there's some other format than those illustrate, that's another issue I couldn't observe with the datafiles I had at hand.

[Brandon Bush]

Upon hindsight the code does work, the only thing is that the first figure still comes up blank

[Brandon Bush]

I managed to fix the issue by adding an extra 5x1 struct for the GOES data. it prints for all days in the range except for the first day

[dpb]

Can you attach a file that seems to have the problem being read? I didn't see that with the ones I looked at. I haven't had time to download and do anything with the .mat files; I'll try to find a few minutes this evening and see if can't clean that up some.

The three Excel files here are all each read as a 31x4 table with variables RG_, Lat, Lon, Accumulation and all data is returned correctly. There's a warning because the RG# column header isn't a valid variable name in ML so it gets converted; I didn't bother to add the extra to remove the warning; that's easy enough to do by not reading the variable names and then assigning them. I used readtable over xlsread simply for performance; xlsread is sloooow!

As for the blank figure, you must have a figure or other plotting command somewhere there without anything shown to it or a full set of data with NaN, maybe...that would be silently ignored by plot.

I didn't have any empties here, but then again couldn't run the whole thing to date...

ADDENDUM I take that back; there is an empty figure; I hadn't ever worried about CLOSEing stuff out so I just thought it was a fignewton of prior failings during debugging...otomh I don't see where that comes from, either...will poke around a little and see what can find.

I presume there's nothing in the coastline except tracing it on top so I'll just comment that out here as don't have it, either, at present.

[Brandon Bush]

I've stepped out of my office for the day. I'll be back in tomorrow morning

[dpb]

OK...

It's colormap that opens the spurious figure; since there isn't an active figure at that point, it has to create one to operate in.

It looks like need to just load the file and then move the colormap call itself to inside the loop...

I didn't get the blank figure first since I didn't have the file I didn't have the call to colormap, either...

[dpb]

"however, the data in the RNFLS variable only comes out to 0"

The two .txt files for 3/16 and 3/17 are all zero values for accumulation so it's not surprising they're read that way... :)

It looks to me things are working as they should from the data files you've posted.

I did just check on the scatter plots; what looks peculiar is that the locations of the points are always the same; it's only the color coding that changes. I did check and it looks to me like that's working as well -- still think it could be more efffectively coded but I'm out of time for tonight.

I don't see anything that seems to agree with the comments above about the data files not being read correctly and needing to do anything to fixup anything regarding them.

The one thing I did discover is that there is a "1" instead of "i" in the title line for the date so that they'll all be labeled the first day in the original plus the above issue about where the call of colormap is--I've fixed those and reattached an updated script.

[dpb]

I looked at the color partitioning scheme some this morning; there are a couple of issues I see...first, the 10 colors don't cover the entire range of the levels vector so you could potentially not count some observations and also the line

idx = find(G.Accum>levels(r) & G.Accum<levels(r+1));

will miss any accumulation values that are identically equal to one of the levels; it should be inclusive on either the upper or lower boundary; your choice as to which.

idx = find(G.Accum>levels(r) & G.Accum<=levels(r+1));

[Brandon Bush]

Upon further testing, I have discovered two issues. The first is that if I extend the range from 3/15/2017 to 4/4/2017, for example, the Gauge structs pulls the available files in that range, however, the files in the GOES struct appears in the exact same order as that in the dn variable instead of the same way as the GAUGE struct, which is in numerical order.

The second is that the "i" int eh for loop always stops at 5 and I only get four plots as an output. The only change I have made was adding GOES = GOES(ismember(dn,datetime(D1:D2))); so that the GOES struct can take the same files as the GAUGE struct.

[Brandon Bush]

Unfortunately I cannot upload anymore files for the time being, I was going to attach some additional data so that you could test an extended range

[Brandon Bush]

update: I was able to fix the numerical order issue for the GOES struct, the only problem that remains is getting the rest of the plots to print out

[dpb]

The two directory structures will EACH be returned in lexical order, but independently.

As noted from the beginning, what this code omits is any error checking to ensure there are matching files in each struct; there are a couple of ways to do this that aren't that difficult; the first obvious check is that they're the same length; the second is to ensure all(dn1==dn2). If those aren't so, then "Houston, we have a problem!".

The alternative way to look at it is to choose one or the other as the "master" set; use dir on it and then try to open the matching other type of the same name. That will ensure the two have the same name as a side effect and you'll find out at run time if there is one missing.

The second two files here don't produce contours owing to, it seems there being bum data that griddata doesn't like in trying to create the 2D response. I've not had time to really delve into just what's going on, but that superficially seems the issue...

[Brandon Bush]

Final Update: I was able to fix the remaining issues and the plots are coming out as intended. I thank you for all of the hard work over the past couple of days

[dpb]

Kewl...glad to hear. I may still play around with that scatter plot part a little...

What is the deal with the levels array vis a vis the 10 bins, though???

[Brandon Bush]

I haven't taken a look at the bins yet because I am working on another part of the project. Once all is said and done I will be able to come back and try to simplify the code using the methods above

[dpb]

Ah-so! I interpreted the "plots as wanted" as thinking were done...I didn't get a round tuit on converting to scatter here, either! :)

Answer 2

[dpb]

Try this as outline to retrieve the files desired between two dates...

D1=datetime("3/15/2017");  D2=datetime("3/16/2017");  % set the desired date range
d=dir('*-*-*.xlsx');                                  % return the list of all files
dn=datetime(extractBefore(cellstr(char(d.name)),'.'),'InputFormat','M-d-yy'); % get datetime array
d=d(ismember(dn,datetime(D1:D2)));                    % save only those between D1, D2
for i=1:length(d)                                     % iterate over those desired only
  g=readtable(d(i).name);                             % read into a table..
  g.RG_=categorical(g.RG_);                           % turn the RG# into categorical
end

For illustration in going forward, the above leaves you with

>> g(1:4,:)
ans =
  4×4 table
     RG_      Lat        Lon      Accumulation
    _____    ______    _______    ____________
    RG002    35.425    -82.971         0      
    RG003    35.385    -82.916         0      
    RG004    35.368     -82.99         0      
    RG005    35.409    -82.965    4.6595      
>>

on each step; I'm not sure precisely what to do inside the loop for each day??? Now, again, this presently is overwriting the variable g ( for "gauge", original huh? :) ) so 'splain zackly what it is that is being plotted; are all of these points on one figure for all times or just what?

If get that clear, then it's probably pretty simple to do what needed...

NB: The D1, D2 values are simply the user-set limits desired; set them in whatever input format is wanted for the UI.

NB2: The same idea works for the .txt files with same naming scheme, of course. You could make the presumption that if you find one set you'll find the other and just open the alternate by changing extension but it's probably more robust to do the directory scan up front and compare first for the user if aren't commensurate sets.

[Brandon Bush]

What's being plotted is the last column from the text files as contours(satellite data) and then the last column from the excel files as points(the rain gauges) on top of the satellite data.

The gauge points have multiple plot functions to make sure that amounts of 0, amounts less than the color bar range and amounts within the color bar range all get plotted.

[dpb]

But what's to be done with each day; a new figure, add to the first or what? I'm not sure where the issue is, precisely about the days.

I guess on re-reading the initial, it isn't to plot a range within the overall set; I thought that's what was the Q?.

But, I still don't know what is to be done by day, precisely...is it just do the same thing for every day with a new figure for each or what?

The first would seem essentially trivial to just add a call to "figure" before beginning to plot; maybe I'm trying to make more out of it than there is???

[Brandon Bush]

Each day should produce an individual plot

[dpb]

OK....I'm beginning to get the picture... :) I was totally off-base from the git-go here of what the issue(s) were, sorry for the distractions.

In that case I will go look at the function; I thought it was basically immaterial to the problem at hand; I think some factoring there may help but probably it can work almost as is...

[Brandon Bush]

No problem, thank you for all your help.