summarize

Summarize object detection performance metrics at data set and class level

Since R2024b

Syntax

[summaryDataset,summaryClass] = summarize(metrics)

[summaryDataset,summaryClass] = summarize(metrics,MetricName=metricName)

Description

[summaryDataset,summaryClass] = summarize(metrics) summarizes the object detection evaluation metrics across all classes of the data set summaryDataset, and for each class in the data set summaryClass.

[summaryDataset,summaryClass] = summarize(metrics,MetricName=metricName) specifies the metrics category to return in the metrics summary.

Examples

Summarize Object Detection Metrics

Load a table containing image file names and ground truth bounding box labels into the workspace. The first column contains the image file names, and the remaining columns contain the labeled bounding boxes.

data = load("vehicleTrainingData.mat");
trainingData = data.vehicleTrainingData;

Set the value of the dataDir variable as the location where the vehicleTrainingData.mat file is located. Load the test data into a local vehicle data folder.

dataDir = fullfile(toolboxdir("vision"),"visiondata");
trainingData.imageFilename = fullfile(dataDir,trainingData.imageFilename);

Create an imageDatastore using the files from the table.

imds = imageDatastore(trainingData.imageFilename);

Create a boxLabelDatastore using the label column from the table.

blds = boxLabelDatastore(trainingData(:,2:end));

Load a pretrained YOLO v2 object detector, trained to detect vehicles, into the workspace.

vehicleDetector = load("yolov2VehicleDetector.mat");
detector = vehicleDetector.detector;

Run the detector on the test images. Set the detection threshold to a low value to detect as many objects as possible. This enables you to evaluate the detector precision across the full range of recall values.

results = detect(detector,imds,Threshold=0.01);

Compute metrics for evaluating the performance of the object detector.

metrics = evaluateObjectDetection(results,blds);

Summarize the object detection metrics for the vehicle class using the summarize object function.

[summaryDataset,summaryClass] = summarize(metrics);
summaryClass

summaryClass=1×3 table
               NumObjects    APOverlapAvg     AP0.5 
               __________    ____________    _______

    vehicle       336          0.99096       0.99096

Input Arguments

metrics — Object detection performance metrics

objectDetectionMetrics object

Object detection performance metrics, specified as an objectDetectionMetrics object.

metricName — Metrics category to return

"all" (default) | "AP" | "LAMR" | "AOS"

Metrics category to return in the metrics summary, specified as one of these options.

• "all" – Returns the average precision and mean average precision metric values, aggregated over each overlap threshold, along with all additional metrics.

• "AP" – Returns only the average precision across all classes at each overlap threshold as a numThresh-by-1 numeric vector, and the mean average precision (mAP) computed across all overlap thresholds. numThresh is the number of overlap thresholds specified by the OverlapThreshold property of the objectDetectionMetrics object.

• "LAMR" – Returns only the log-average miss rate (LAMR) computed at each overlap threshold as a numThresh-by-1 numeric vector, and the mean of the log-average miss rate (mLAMR) computed across all overlap thresholds.

• "AOS" – Returns only the average orientation similarity (AOS) computed at each overlap threshold as a numThresh-by-1 array, and the mean of the average orientation similarity (mAOS) computed across all overlap thresholds.

  To specify "AOS" as the metric to return, your input data must contain rotated bounding boxes.

Output Arguments

summaryDataset — Metrics summary over data set

table

Metric summary over the data set, returned as a 1-by-numMetrics table, where numMetrics is the number of specified metrics.

The columns of the table depend on the value of the metricName argument.

• "all" —

  NumObjects	mAPOverlapAvg	mAP	mLAMROverlapAvg	mLAMR	mAOSOverlapAvg	mAOS
  Number of objects in the ground truth data, returned as a positive integer.	Mean average precision (mAP) averaged over all thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Mean average precision (mAP), or average precision averaged over all classes, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector. numThresh is the number of overlap thresholds.	Mean log-average miss rate (mLAMR) averaged over all classes and all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Mean log-average miss rate (mLAMR) averaged over all classes, computed at all overlap thresholds, specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.	Mean average orientation similarity (mAOS) averaged over all classes, and averaged over all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Mean average orientation similarity (mAOS) averaged over all classes, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.

  To return the "mAOSOverlapAvg" and "mAOS" columns, your input data must contain rotated bounding boxes.

• "AP" —

  NumObjects	mAPOverlapAvg	mAP
  Number of objects in the ground truth data, returned as a positive integer.	Mean average precision (mAP) averaged over all thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Mean average precision (mAP), or average precision averaged over all classes, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector. numThresh is the number of overlap thresholds.

• "LAMR" —

  mLAMROverlapAvg	mLAMR
  Mean log-average miss rate (mLAMR) averaged over all classes and all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a positive scalar.	Mean log-average miss rate (mLAMR) averaged over all classes, computed at all overlap thresholds, specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.

• "AOS" —

  mAOSOverlapAvg	mAOS
  Mean average orientation similarity (mAOS) averaged over all classes and all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Mean average orientation similarity (mAOS) averaged over all classes, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.

summaryClass — Metrics summary per class

table

Metrics summary per class, returned as a numClasses-by-numMetrics table, where numClasses is the number of classes, and numMetrics is the number of specified metrics.

The columns of the table depend on the value of the metricName argument.

• "all" —

  NumObjects	APOverlapAvg	AP	LAMROverlapAvg	LAMR	AOSOverlapAvg	AOS
  Number of objects in the ground truth data for each class, returned as a positive integer.	Average precision (AP) for each class averaged over all thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Average precision (AP) for each class, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector. numThresh is the number of overlap thresholds.	Log-average miss rate (LAMR) for each class, averaged over all overlap thresholds specified by the OverlapThreshold property of metrics.	Log-average miss rate (LAMR) for each class, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.	Average orientation similarity (AOS) for each class, averaged over all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Average orientation similarity (AOS) for each class, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.

  To return the "AOSOverlapAvg" and "AOS" columns, your input data must contain rotated bounding boxes.

• "AP" —

  NumObjects	APOverlapAvg	AP
  Number of objects in the ground truth data for each class, returned as a positive integer.	Average precision (AP) for each class averaged over all thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Average precision (AP) for each class, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector. numThresh is the number of overlap thresholds.

• "LAMR" —

  LAMROverlapAvg	LAMR
  Log-average miss rate (LAMR) for each class, averaged over all overlap thresholds specified by the OverlapThreshold property of metrics.	Log-average miss rate (LAMR) for each class, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.

• "AOS" —

  AOSOverlapAvg	AOS
  Average orientation similarity (AOS) for each class, averaged over all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numeric scalar.	Average orientation similarity (AOS) for each class, computed at all overlap thresholds specified by the OverlapThreshold property of metrics, returned as a numThresh-by-1 numeric vector.

Version History

Introduced in R2024b