Trying to use retime to sum daily values, but I don't want days that only have NaN values to become 0

Basically I have a timetable or rainfall values (a column with half-hourly time increments and a column with the rainfall at each time point). I used retime to get daily sums:

sum_rainfall_2018 = retime(rainfall_2018, 'daily','sum');

This mostly works, but I have a number of days that only have NaN values, and retime is summing those days as 0. I need them to stay NaN because I need to know where my gaps are. Is there a simple way to do this?

Answer 1

[Jonas]

weird, the normal behavior of sum is giving NaN if at least one of the summed values is NaN. try @sum instead of 'sum' and if this also does not work, use a self defined function which forces the exact output

Answer 2

[Peter Perkins]

In the doc for retime (admittedly, not really in flashing bold letters): "All the listed methods omit NaNs, NaTs, and other missing data indicators, except for func. To include missing data indicators, specify func as a function handle to a function that includes them when aggregating data."

Haven't tried it, but I believe that if you pass in @sum, not 'sum', you will get what you want.

[Everett Snieder]

Doesn't work

Answer 3

[Everett Snieder]

OK, I figured it out... basically when you pass an empty array into sum(), it returns 0. So to handle big time gaps (larger than your timestep), you need to create a special sum() function and pass the handle to retime:

function y = sum2(x)
if isempty(x)
    y = nan();
else
    y = sum(x);
end

[Peter Perkins]

Right, prod and sum return the "unitary element" when given an empty:

prod([])

ans = 1

sum([])

ans = 0

But I don't think you need to do this:

tt = timetable([1;2;NaN;4;NaN;NaN],'RowTimes',datetime(2022,3,[21;21;22;22;23;23]))

tt = 6×1 timetable
       Time        Var1
    ___________    ____

    21-Mar-2022      1 
    21-Mar-2022      2 
    22-Mar-2022    NaN 
    22-Mar-2022      4 
    23-Mar-2022    NaN 
    23-Mar-2022    NaN 

retime(tt,'daily','sum') % 'sum' removes NaNs

ans = 3×1 timetable
       Time        Var1
    ___________    ____

    21-Mar-2022     3  
    22-Mar-2022     4  
    23-Mar-2022     0  

retime(tt,'daily',@sum) % @sum does not

ans = 3×1 timetable
       Time        Var1
    ___________    ____

    21-Mar-2022      3 
    22-Mar-2022    NaN 
    23-Mar-2022    NaN 

retime(tt,'daily',@(x)sum(x,'omitnan')) % equivalent to 'sum'

ans = 3×1 timetable
       Time        Var1
    ___________    ____

    21-Mar-2022     3  
    22-Mar-2022     4  
    23-Mar-2022     0  

It may be that your desired behavior is, "remove NaNs before summing, unless all NaN, in which case return NaN." For that, you would need your own function, because the standard behavior of sum on empty is to return 0.