Outlier Detection of a matrix depending on spatio coordinates
6 views (last 30 days)
Show older comments
Hi, I have thousands of matrices, each of (130x160) with values (range of -25 up to +25) consisting of outliers.The values of the matrix depending on spatial data x(1:130) and Y(1:160). Outliers are often within the range and sometimes not only single values, clustering across 2 or 3 values across the raws and/or columns (X or Y) of the matrix.
I already tried "filloutliers" and deleteoutliers (https://de.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers)
with rather average level of satisfaction.
Can anyone recommend an appropriate method to detect the outliers? What kind of ML algorithms could I try?
Thanks,
Chris
14 Comments
Alan
on 4 Sep 2023
I had intended to post my answer in the "Answers" section instead of the "Coments" section. Therefore I shall repost it in the "Answers" section. I hope that the OP and others looking for a solution to a similar problem will find the resources to be helpful.
Accepted Answer
Alan
on 4 Sep 2023
Edited: Alan
on 4 Sep 2023
(This is a repost from the comments section)
If you want to detect outliers within each matrix rather than considering each matrix as datapoints.
In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://www.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2
Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:
x = 1:130;
y = 1:160;
[X Y] = meshgrid(x, y);
% Generating a matrix for this example
Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));
% Fitting surface
xdata = reshape(X, 1, [])';
ydata = reshape(Y, 1, [])';
zdata = reshape(Z, 1, [])';
fitted_surface = fit([xdata ydata], zdata, "poly23");
% Visualize fitted surface against the matrix
plot(fitted_surface, [xdata ydata], zdata)
legend(["Fitted Surface"]);
% Continue to calculate z-score and exclude outliers
In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:
- isoutlier() - https://www.mathworks.com/help/matlab/ref/isoutlier.html
- rmoutliers() - https://www.mathworks.com/help/matlab/ref/rmoutliers.html
- filoutliers() - https://www.mathworks.com/help/matlab/ref/filloutliers.html
Some parameters to experiment with could be:
- dim : Dimension along which the outliers should be detected.
- movemethod : To detect outliers using a moving window.
- percentiles : To identify the extreme percentile of data.
As for machine learning methods, you could check out this resource: https://www.mathworks.com/help/stats/unsupervised-anomaly-detection.html . It contains various supervised methods for multivariate data.
0 Comments
More Answers (0)
See Also
Categories
Find more on Statistics and Machine Learning Toolbox in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!