How to "teach" Matlab how to remove inconsistent data?

Hello!

I have a set of data consisting of measured values in time, over several days. You can see on the plot below the "normal" shape, and the moments at which data is inconsistent (mostly in the left and right parts of the plot). Note that this is values from one of my measurement devices, other ones have a more consistent shape with values decreasing at the beginning and at the end. I would like to remove this inconsistent data. How can I do this? I thought of doing some machine learning, for example by training the algorithm and teaching it what is "good" or "bad" data.

Thank you

Romain

Answer 1

[Walter Roberson]

http://www.mathworks.com/matlabcentral/fileexchange/34795-outlier-detection-and-removal--hampel-

http://www.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers

http://www.mathworks.com/matlabcentral/answers/121247-how-can-i-detect-and-remove-outliers-from-a-large-dataset

Answer 2

[Star Strider]

One possibility is for you to sort your data by the independent variable and then plot it. That might at least make it a bit easier to understand. You could then decide what the outliers are.

[Romain Meyer]

Thank you both for your answers. Unfortunately I'm more interested in the shape of the time series, because the 100% values are not all outliers. I want to remove data if the detector seems to behave anormaly, but not based on the values.

Romain

[Star Strider]

‘... if the detector seems to behave anormaly ...’

How do you determine that?

[Romain Meyer]

The values should increase gradually, or at least be consistent with the time and the day. For example on the picture, it is clear that the values are not valid before 7am and after 9pm for some days. I know this is very complicated to implement that kind of verification. The idea I had is to have a script that present some time series and wait for the user "OK" or "NOT OK" and then it could learn to detect bad data, based on the training.

[Star Strider]

I’m not quite sure what you’re measuring, but it would seem that 100% occupancy aren’t correct data. I would remove those (perhaps anything greater than 95%, and the associated time data), and then see what the data are. You might not need anything more sophisticated than thresholding your data.