Remove noise from time series data

Hello Matlab community,

I have a time series data from different levels with 30 min interval. I tried to use the medfilt1 to remove outliers and worked at some level.

data2=medfilt1(data1,3)

Here is the plot of data1:

And here is the plot of data2:

However what I want to remove is the red crossed parts of the each level. I marked the purple data as an example. I do not want to make a perfectly smooth data but only to remove fluctuations exist at some points.

I wanted to apply 3 sigma method as an option and integrated the code of Adam Danz' code as below but I should have done something missing.

m = mean(data2); 
sd = std(data2); 
outliers = false(size(data2)); 
outliers(data2 < m-sd*3) = true; 
outliers(data2 > m+sd*3) = true; 
figure
t = 1:length(data2); 
plot(t, data2, 'b.')
hold on
plot(t(outliers), d(outliers), 'ro')
rh = refline(0,m); 
set(rh, 'color', 'm')
rh2 = refline(0,m+sd*3); 
rh3 = refline(0,m-sd*3); 
set([rh2,rh3], 'color', 'm', 'linestyle', '--')
legend('data', 'outliers', 'mean', '3rd sd')

All I want to do is to filter the fluctiations for each level but I really get confused about what I am doing.

I will glad to hear your suggestions.

Best,

Ezgi

Answer 1

[Star Strider]

The sgolayfilt function might be a better option.

Try something like this —

FrameLen = 201;

DataFilt = sgolayfilt(data1, 3, FrameLen);

figure

plot(t, DataFilt)

grid

Experiment with various values of ‘FrameLen’ to get the result you want.

.

[Cakil]

Thank you, it was a good alternative.

I will be happy to hear if there is another option to smooth only the outlier data by filtering but not the whole dataset.

[Star Strider]

My choice would be the filloutliers funciton. It will work on matrix columns, and works here. You will have to choose the appropriate options to get the desired result.

One approach that seems to work for me:

DataFilt = filloutliers(data1, 'center','movmedian', 250);

figure

subplot(2,1,1)

plot(data1)

grid

subplot(2,1,2)

plot(DataFilt)

grid

See the documentation for details, and other options to experiment with (there are several).

.

[Cakil]

Thank you for your support, it was very helpful.

[Star Strider]

As always, my pleasure!