accuracyMetric

Deep learning accuracy metric

Since R2023b

Description

Use an AccuracyMetric object to track the network accuracy when you train or test a deep neural network.

To specify which metrics to use during training, specify the Metrics option of the trainingOptions function. You can use this option only when you train a network using the trainnet function.

To plot the metrics during training, in the training options, specify Plots as "training-progress". If you specify the ValidationData training option, then the software also plots and records the metric values for the validation data. To output the metric values to the Command Window during training, in the training options, set Verbose to true.

You can also access the metrics after training using the TrainingHistory and ValidationHistory fields from the second output of the trainnet function.

To specify which metrics to use when you test a neural network, use the metrics argument of the testnet function.

Creation

Syntax

metric = accuracyMetric

metric = accuracyMetric(Name=Value)

Description

metric = accuracyMetric creates an AccuracyMetric object. You can then specify metric as the Metrics name-value argument in the trainingOptions function or the metrics argument of the testnet function. With no additional options specified, this syntax is equivalent to specifying the metric as "accuracy".

This metric is valid only for classification tasks.

metric = accuracyMetric(Name=Value) sets the Name, NumTopKClasses (since R2024b), NetworkOutput, AverageType, and ClassificationMode properties using name-value arguments.

Properties

Name — Metric name

"Accuracy" (default) | string scalar | character vector

Metric name, specified as a string scalar or character vector. The metric name appears in the training plot, the verbose output, the training information that you can access as the second output of the trainnet function, and table output of the testnet function.

Data Types: char | string

NumTopKClasses — Number of top classes to consider

1 (default) | positive integer

Since R2024b

This property is read-only.

Number of top classes (k) to consider when computing the top-k accuracy, specified as a positive integer. The NumTopKClasses value must be less than or equal to the number of classes. For more information, see Top-K Accuracy.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

NetworkOutput — Name of layer to apply metric to

[] (default) | string scalar | character vector

This property is read-only.

Name of the layer to apply the metric to, specified as [], a string scalar, or a character vector. When the value is [], the software passes all of the network outputs to the metric.

Note

You can apply the built-in metric to only a single output. If you have a network with multiple outputs, then you must specify the NetworkOutput name-value argument. To apply built-in metrics to multiple outputs, you must create a metric object for each output.

Data Types: char | string

AverageType — Type of averaging to use

"micro" (default) | "macro" | "weighted"

This property is read-only.

Type of averaging to use to compute the metric, specified as one of these values:

• "micro" — Calculate the metric across all classes.

• "macro" — Calculate the metric for each class and return the average.

• "weighted" — Calculate the metric for each class and return the weighted average. The weight for a class is the proportion of observations from that class.

For more information, see Averaging Type.

Data Types: char | string

ClassificationMode — Type of classification task

"single-label" (default) | "multilabel"

This property is read-only.

Type of classification task, specified as one of these values:

• "single-label" — Each observation is exclusively assigned one class label (single-label classification).

• "multilabel" — Each observation can be assigned more than one independent class label (multilabel classification). The software uses a softmax threshold of 0.5 to assign class labels.

To select the classification mode for binary classification, consider the final layer of the network:

• If the final layer has an output size of one, such as with a sigmoid layer, use "multilabel".

• If the final layer has an output size of two, such as with a softmax layer, use "single-label".

Note

This metric is not supported when the ClassificationMode is set to "single-label" and the network output has a channel dimension of size 1. For example, if you have a single class and the output is a sigmoidLayer object (binary-sigmoid task).

Data Types: char | string

Maximize — Flag to maximize metric

1 (true)

This property is read-only.

Flag to maximize metric, specified as 1 (true) if the optimal value for the metric occurs when the metric is maximized.

For this metric, the Maximize value is always set to 1 (true).

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | logical

Object Functions

trainingOptions	Options for training deep learning neural network
trainnet	Train deep learning neural network

Examples

Plot and Record Accuracy During Training

Plot and record the training and validation accuracy when you train a deep neural network.

Unzip the digit sample data and create an image datastore. The imageDatastore function automatically labels the images based on folder names.

unzip("DigitsData.zip")
imds = imageDatastore("DigitsData", ...
    IncludeSubfolders=true, ...
    LabelSource="foldernames");

The datastore contains 10,000 synthetic images of digits from 0 to 9. Each image in the data set has a size of 28-by-28-by-1 pixels. You can train a deep learning network to classify the digit in the image.

Use a subset of the data as the validation set.

numTrainingFiles = 750;
[imdsTrain,imdsVal] = splitEachLabel(imds,numTrainingFiles,"randomize");

Create an image classification network.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    maxPooling2dLayer(2,Stride=2)
    fullyConnectedLayer(10)
    softmaxLayer];

Create an AccuracyMetric object and set AverageType to "macro". You can use this object to record and plot the training and validation accuracy.

metric = accuracyMetric(AverageType="macro")

metric = 
  AccuracyMetric with properties:

                  Name: "Accuracy"
           AverageType: "macro"
    ClassificationMode: "single-label"
        NumTopKClasses: 1
         NetworkOutput: []
              Maximize: 1

Specify the accuracy metric in the training options. To plot the accuracy during training, set Plots to "training-progress". To output the values during training, set Verbose to true.

options = trainingOptions("adam", ...
    MaxEpochs=5, ...
    Metrics=metric, ...
    ValidationData=imdsVal, ...
    ValidationFrequency=50, ...
    Plots="training-progress", ...
    Verbose=true);

Train the network using the trainnet function.

[net,info] = trainnet(imdsTrain,layers,"crossentropy",options);

    Iteration    Epoch    TimeElapsed    LearnRate    TrainingLoss    ValidationLoss    TrainingAccuracy    ValidationAccuracy
    _________    _____    ___________    _________    ____________    ______________    ________________    __________________
            0        0       00:00:03        0.001                            13.488                                      9.84
            1        1       00:00:04        0.001          13.974                                7.2727                      
           50        1       00:00:22        0.001          2.7428            2.7443              67.098                    65
          100        2       00:00:39        0.001           1.291            1.2181              74.996                 79.08
          150        3       00:00:53        0.001         0.64684           0.79978              87.382                 84.04
          200        4       00:01:07        0.001         0.18809           0.53245              94.948                 88.88
          250        5       00:01:22        0.001         0.14852           0.49394               95.06                 89.36
          290        5       00:01:34        0.001          0.2689            0.3885              94.064                 91.48
Training stopped: Max epochs completed

Access the loss and accuracy values for the validation data.

info.ValidationHistory

ans=7×3 table
    Iteration     Loss      Accuracy
    _________    _______    ________

         0        13.488      9.84  
        50        2.7443        65  
       100        1.2181     79.08  
       150       0.79978     84.04  
       200       0.53245     88.88  
       250       0.49394     89.36  
       290        0.3885     91.48  

Compare Top-K Accuracy

Since R2024b

Compare the top-1, top-2, and top-3 accuracy.

Unzip the digit sample data and create an image datastore.

unzip("DigitsData.zip")
imds = imageDatastore("DigitsData", ...
    IncludeSubfolders=true, ...
    LabelSource="foldernames");

Use a subset of the data as the validation set.

numTrainingFiles = 750;
[imdsTrain,imdsVal] = splitEachLabel(imds,numTrainingFiles,"randomize");

Create an image classification network.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    maxPooling2dLayer(2,Stride=2)
    fullyConnectedLayer(10)
    softmaxLayer];

Create an AccuracyMetric object for each of the top-k accuracy metrics.

acc1 = accuracyMetric(NumTopKClasses=1,Name="Top1Accuracy");
acc2 = accuracyMetric(NumTopKClasses=2,Name="Top2Accuracy");
acc3 = accuracyMetric(NumTopKClasses=3,Name="Top3Accuracy");

Specify the accuracy metrics in the training options. To plot the metrics during training, set Plots to "training-progress".

options = trainingOptions("adam", ...
    MaxEpochs=5, ...
    Metrics=[acc1,acc2,acc3], ...
    ValidationData=imdsVal, ...
    ValidationFrequency=50, ...
    Plots="training-progress", ...
    Verbose=false);

Train the network using the trainnet function.

[net,info] = trainnet(imdsTrain,layers,"crossentropy",options);

Compare the final top-k metrics for the validation data.

finalValidation = info.ValidationHistory(end,["Top1Accuracy" "Top2Accuracy" "Top3Accuracy"]);

bar(table2array(finalValidation))
xticklabels(finalValidation.Properties.VariableNames);
ylabel("Accuracy (%)")

More About

Top-K Accuracy

Top-k accuracy measures if the true class label is within the labels that correspond with the top k prediction scores. For example, if k is 3, then a prediction is classified as accurate if the true class label is in the top three class scores labels predicted by the model.

Averaging Type

When you compute a metric, you can aggregate the results into a single value in several ways.

• Micro average — Combine contributions of all observations from each class. For example, calculate the true positive (TP), false positive (FP), true negative (TN), and false negative (FN) across all C classes and then compute the metric.

• Macro average — Calculate the metric for each class and then take the average across all the classes. For example, use the true positive rate for just those observations in class i (TP_i).

• Weighted average — Calculate the metric for each class and then take the weighted average.

This table shows the accuracy formulation for different options.

Accuracy Formulas

Single-Label		Multilabel
Micro Averaging	Macro or Weighted Averaging	Micro Averaging	Macro or Weighted Averaging
$$\text{accuracy}=\frac{TP}{TP\text{}+FN}$$	$$\text{accuracy}={\displaystyle \sum _{i=1}^C{w}_i\frac{T{P}_i}{T{P}_i+F{N}_i}}$$	$$\text{accuracy = }\frac{TP+TN}{TP+TN+FP+FN}$$	$$\text{accuracy}={\displaystyle \sum _{i=1}^K{w}_i\frac{T{P}_i+T{N}_i}{T{P}_i+T{N}_i+F{P}_i+F{N}_i}}$$

The weight w_i controls the weighting for each class in macro and weighted averaging according to these formulas.

• Macro — $$w_i=\frac{1}{C}$$

• Weighted — $$w_i=\frac{n_i}{n}$$, where n_i is the number of observations in class i and n is the total number of observations.

Version History

Introduced in R2023b

R2024b: Compute Top-k Accuracy

The AccuracyMetric object now has the NumTopKClasses property. You can use this property to compute the top-k accuracy for any valid k value. Before R2024b, only top-1 accuracy was supported.