Main Content

isanomaly

Find anomalies in data using robust random cut forest (RRCF) for incremental learning

Since R2023b

    Description

    tf = isanomaly(forest,Tbl) finds anomalies in the table Tbl using the incrementalRobustRandomCutForest object forest and returns the logical array tf, whose elements are true when an anomaly is detected in the corresponding row of Tbl. You must use this syntax if you train forest by passing a table to fit, or if you create forest using the incrementalLearner function with a RobustRandomCutForest object trained on data in a table.

    example

    tf = isanomaly(forest,X) finds anomalies in the matrix X. You must use this syntax if you train forest by passing a matrix to fit, or if you create forest using the incrementalLearner function with a RobustRandomCutForest object trained on data in a matrix.

    tf = isanomaly(___,ScoreThreshold=scoreThreshold) specifies the threshold for the anomaly score using any of the input argument combinations in the previous syntaxes. isanomaly identifies observations with scores above scoreThreshold as anomalies.

    [tf,scores] = isanomaly(___) also returns an anomaly score in the range [0,Inf) for each observation in Tbl or X. A small positive value indicates a normal observation, and a large positive value indicates an anomaly.

    Examples

    collapse all

    Train a robust random cut forest (RRCF) model on a simulated, noisy, periodic shingled time series containing no anomalies by using rrcforest. Convert the trained model to an incremental learner object, and then incrementally fit the time series and detect anomalies.

    Create Simulated Data Stream

    Create a simulated data stream of observations representing a noisy sinusoid signal.

    rng(0,"twister"); % For reproducibility
    period = 100;
    n = 2001+period;
    sigma = 0.04;
    a = linspace(1,n,n)';
    b = sin(2*pi*(a-1)/period)+sigma*randn(n,1);

    Introduce an anomalous region into the data stream. Plot the data stream portion that contains the anomalous region, and circle the anomalous data points.

    c = 2*(sin(2*pi*(a-35)/period)+sigma*randn(n,1));
    b(1150:1170) = c(1150:1170);
    scatter(a,b,".")
    xlim([900,1200])
    xlabel("Observation")
    hold on
    scatter(a(1150:1170),b(1150:1170),"r")
    hold off

    Figure contains an axes object. The axes object with xlabel Observation contains 2 objects of type scatter.

    Convert the single-featured data set b into a multi-featured data set by shingling [1] with a shingle size equal to the period of the signal. The ith shingled observation is a vector of k features with values bi, bi+1, ..., bi+k-1, where k is the shingle size.

    X = [];
    shingleSize = period;
    for i = 1:n-shingleSize
        X = [X;b(i:i+shingleSize-1)'];
    end

    Train Model and Perform Incremental Anomaly Detection

    Fit a robust random cut forest model to the first 1000 shingled observations, specifying a contamination fraction of 0. Convert the model to an incrementalRobustRandomCutForest model object. Specify to keep the 100 most recent observations relevant for anomaly detection.

    Mdl = rrcforest(X(1:1000,:),ContaminationFraction=0);
    IncrementalMdl = incrementalLearner(Mdl,NumObservationsToKeep=100);

    To simulate a data stream, process the full shingled data set in chunks of 100 observations at a time. At each iteration:

    • Process 100 observations.

    • Calculate scores and detect anomalies using the isanomaly function.

    • Store anomIdx, the indices of shingled observations marked as anomalies.

    • If the chunk contains fewer than three anomalies, fit and update the previous incremental model.

    n = numel(X(:,1));
    numObsPerChunk = 100;
    nchunk = floor(n/numObsPerChunk);
    anomIdx = [];
    allscores = [];
    
    % Incremental fitting
    rng("default"); % For reproducibility
    for j = 1:nchunk
        ibegin = min(n,numObsPerChunk*(j-1) + 1);
        iend = min(n,numObsPerChunk*j);
        idx = ibegin:iend;
        [isanom,scores] = isanomaly(IncrementalMdl,X(idx,:));
        allscores = [allscores;scores];
        anomIdx = [anomIdx;find(isanom)+ibegin-1];
        if (sum(isanom) < 3)
            IncrementalMdl = fit(IncrementalMdl,X(idx,:));
        end
    end

    Analyze Incremental Model During Training

    At each iteration, the software calculates a score value for each observation in the data chunk. A negative score value with large magnitude indicates a normal observation, and a large positive value indicates an anomaly. Plot the anomaly score for the observations in the vicinity of the anomaly. Circle the scores of shingles that the software returns as anomalous.

    figure
    scatter(a(1:2000),allscores,".")
    hold on
    scatter(a(anomIdx),allscores(anomIdx),20,"or")
    xlim([900,1200])
    xlabel("Shingle")
    ylabel("Score")
    hold off

    Figure contains an axes object. The axes object with xlabel Shingle, ylabel Score contains 2 objects of type scatter.

    Because the introduced anomalous region begins at observation 1150, and the shingle size is 100, shingle 1051 is the first to show a high anomaly score. Some shingles between 1050 and 1170 have scores lying just below the anomaly score threshold, due to the noise in the sinusoidal signal. The shingle size affects the performance of the model by defining how many subsequent consecutive data points in the original time series the software uses to calculate the anomaly score for each shingle.

    Plot the unshingled data and highlight the introduced anomalous region. Circle the observation number of the first element in each shingle returned by that the software as anomalous.

    figure
    xlim([900,1200])
    ylim([-1.5 2])
    rectangle(Position=[1150 -1.5 20 3.5],FaceColor=[0.9 0.9 0.9], ...
        EdgeColor=[0.9 0.9 0.9])
    hold on
    scatter(a,b,".")
    scatter(a(anomIdx),b(anomIdx),20,"or")
    xlabel("Observation")
    hold off

    Figure contains an axes object. The axes object with xlabel Observation contains 3 objects of type rectangle, scatter.

    Create a simulated data stream of observations representing a noisy sinusoid signal. Incrementally fit the time series with a robust random cut forest (RRCF) model and detect anomalies.

    rng(0,"twister"); % For reproducibility
    period = 100;
    n = 1000;
    sigma = 0.3;
    a = linspace(1,n,n)';
    X1 = sin(2*pi*a/period)+sigma*randn(n,1);
    X2 = sin(2*pi*a/period/3)+sigma*randn(n,1);

    Introduce an anomalous region into the data stream.

    c = 10*sin(2*pi*(a-20)/period);
    X1(551:570) = c(551:570);
    X2(551:570) = c(551:570);
    X = [X1 X2];

    Create a default incrementalRobustRandomCutForest model object. Specify a score warmup period of 500 observations.

    scoreWarmupPeriod = 500;
    IncrementalMdl = incrementalRobustRandomCutForest(ScoreWarmupPeriod=scoreWarmupPeriod);

    To simulate a data stream, process the full data set in chunks of 100 observations at a time. At each iteration:

    • Process 100 observations.

    • If the incremental model is warm, calculate scores and detect anomalies using the isanomaly function.

    • Store allscores, the scores of the observations.

    • Store anomIdx, the indices of observations marked as anomalies.

    • If the chunk contains fewer than three anomalies, fit and update the previous incremental model.

    numObsPerChunk = 100;
    nchunk = floor(n/numObsPerChunk);
    anomIdx = [];
    allscores = [];
    isanom = [];
    
    % Incremental fitting
    for j = 1:nchunk
        ibegin = min(n,numObsPerChunk*(j-1) + 1);
        iend = min(n,numObsPerChunk*j);
        idx = ibegin:iend;
        if (IncrementalMdl.IsWarm)
            [isanom,scores] = isanomaly(IncrementalMdl,X(idx,:));
            allscores = [allscores;scores];
            anomIdx = [anomIdx;find(isanom)+ibegin-1];
        end    
        if (sum(isanom) < 3)
            IncrementalMdl = fit(IncrementalMdl,X(idx,:));
        end
    end

    Plot the scores for observations after the warmup period. Circle the detected anomalies and indicate the introduced anomalous observations with an x marker.

    scatter(a(scoreWarmupPeriod+1:end),allscores(1:end),".")
    xlabel("Observation")
    ylabel("Score")
    hold on
    scatter(a(551:570), ...
        allscores(551-scoreWarmupPeriod:570-scoreWarmupPeriod),90,"x")
    scatter(a(anomIdx),allscores(anomIdx-scoreWarmupPeriod),20,"or")
    hold off

    Figure contains an axes object. The axes object with xlabel Observation, ylabel Score contains 3 objects of type scatter.

    The software identified all of the observations in the introduced anomalous region as anomalies. However, it also identified several other observations as anomalies due to the noisy sinusoid signal.

    Detect Anomalies Using a Score Threshold Buffer

    Repeat the incremental anomaly detection procedure with a new default incremental RRCF model. Specify a score warmup period of 500 observations. Only observations with scores above ScoreThreshold + thresholdBuffer are indicated as anomalies. Specify thresholdBuffer = 1.

    thresholdBuffer = 1;
    scoreWarmupPeriod = 500;
    IncrementalMdl = incrementalRobustRandomCutForest(ScoreWarmupPeriod=scoreWarmupPeriod);
    numObsPerChunk = 100;
    nchunk = floor(n/numObsPerChunk);
    anomIdx = [];
    allscores = [];
    isanom = [];
    
    % Incremental fitting
    for j = 1:nchunk
        ibegin = min(n,numObsPerChunk*(j-1) + 1);
        iend = min(n,numObsPerChunk*j);
        idx = ibegin:iend;
        if (IncrementalMdl.IsWarm)
            [isanom,scores] = isanomaly(IncrementalMdl,X(idx,:), ...
                ScoreThreshold=IncrementalMdl.ScoreThreshold+thresholdBuffer);
            allscores = [allscores;scores];
            anomIdx = [anomIdx;find(isanom)+ibegin-1];
        end    
        if (sum(isanom) < 3)
            IncrementalMdl = fit(IncrementalMdl,X(idx,:));
        end
    end

    Plot the scores for observations after the warmup period. The scores are different from the previous case due to the stochastic behavior of the RRCF training algorithm. Circle the detected anomalies and indicate the introduced anomalous observations with an x marker.

    scatter(a(scoreWarmupPeriod+1:end),allscores(1:end),".")
    xlabel("Observation")
    ylabel("Score")
    hold on
    scatter(a(551:570), ...
        allscores(551-scoreWarmupPeriod:570-scoreWarmupPeriod),90,"x")
    scatter(a(anomIdx),allscores(anomIdx-scoreWarmupPeriod),20,"or")
    hold off

    Figure contains an axes object. The axes object with xlabel Observation, ylabel Score contains 3 objects of type scatter.

    The software identifies only observations in the introduced anomalous region as anomalies.

    Input Arguments

    collapse all

    Trained incremental RRCF model, specified as an incrementalRobustRandomCutForest model object.

    Predictor data, specified as a table. Each row of Tbl corresponds to one observation, and each column corresponds to one predictor variable. Multicolumn variables and cell arrays other than cell arrays of character vectors are not allowed.

    If you train forest using a table, then you must provide predictor data by using Tbl, not X. All predictor variables in Tbl must have the same variable names and data types as those in the training data. However, the column order in Tbl does not need to correspond to the column order of the training data.

    Data Types: table

    Predictor data, specified as a numeric matrix. Each row of X corresponds to one observation, and each column corresponds to one predictor variable.

    If you train forest using a matrix, then you must provide predictor data by using X, not Tbl. The variables that make up the columns of X must have the same order as the variables in the training data.

    Data Types: single | double

    Threshold for the anomaly score, specified as a numeric scalar in the range [0,Inf). isanomaly identifies observations with scores above the threshold as anomalies.

    The default value is the ScoreThreshold property value of forest.

    Example: ScoreThreshold=0.5

    Data Types: single | double

    Output Arguments

    collapse all

    Anomaly indicators, returned as a logical column vector. An element of tf is true when the observation in the corresponding row of Tbl or X is an anomaly, and false otherwise. tf has the same length as Tbl or X.

    isanomaly identifies observations with scores above the threshold (the ScoreThreshold value) as anomalies.

    Note

    isanomaly assigns the anomaly indicator of false (logical 0) to observations that have missing values for all predictors.

    Anomaly scores, returned as a numeric column vector whose values are in the range [0,Inf). scores has the same length as Tbl or X, and each element of scores contains an anomaly score for the observation in the corresponding row of Tbl or X. A small positive value indicates a normal observation, and a large positive value indicates an anomaly.

    Note

    isanomaly assigns the anomaly score of NaN to observations that have missing values for all predictors.

    References

    [1] Guha, Sudipto, N. Mishra, G. Roy, and O. Schrijvers. "Robust Random Cut Forest Based Anomaly Detection on Streams," Proceedings of The 33rd International Conference on Machine Learning 48 (June 2016): 2712–21.

    Version History

    Introduced in R2023b