how to remove data discontinuities, jumps, outliers, and other artifacts that are not noise. Smoothing is bad,

I'd like to hear of cool ways to process what should nominally be smooth data and detect and remove jumps, single point outliers, and other artifacts that are not noise. The data must be knit together prior to doing a nonlinear regression fit to a model.

Answer 1

[Walter Roberson]

You do not know what is noise and what is not until you have a model to compare against. You therefore cannot eliminate the noise in order to fit to determine what your model is.

With a non-linear model, a value that is much different than the others is not necessarily an artifact or noise. For example it might turn out that the model was y = 1/(a*x-b)^2 in which case if x turned out to be very close to b/a then the y value could be quite big but just a very short distance away y might be much smaller. Eliminating the large y value would, in such a case, lead to an incorrect model.

[Eugene Davis]

i actually know the parametric form for the data. It is a detection system and sometimes there are step changes or single point outliers after which the system returns to the nominal smooth form. Of course trying to model the data without the correction results in an incorrect solution. CUrrently I do detection and correction and it works well. I just wonder what other cool methods folks might have.

[Walter Roberson]

You might know the parametric form, but we do not know it. In the general case with non-linear systems, you cannot know what the noise is until you have produced a complete model with coefficients.

You want to do "outlier detection" to find points that cannot be produced by any combination of parameters for the form of model that you have. That requires a bunch of numeric analysis.

Perhaps the following might help:

https://www.mathworks.com/matlabcentral/fileexchange/34795-outlier-detection-and-removal--hampel-

https://www.mathworks.com/matlabcentral/fileexchange/37211-remove-outliers

https://www.mathworks.com/matlabcentral/fileexchange/28501-tests-to-identify-outliers-in-data-series http://www.mathworks.com/help/signal/ref/hampel.html

Answer 2

[Image Analyst]

Let's say you have a step and you remove it. What will be in it's place? It can't be a step anymore because you removed it. So it will be smoother. It must be because there is no longer a step there. How can you remove a step without smoothing the data? Please show an illustration of that.

[Eugene Davis]

The step removal process knits the data back together. For a step I estimate the place and size of the step and then shift the data to the right of the step by the estimated size.