Interactively Segment and Analyze ROI Using MedSAM and Radiomics

Since R2024b

This example shows how to perform interactive segmentation of an object in a region of interest (ROI) of an image using the Medical Segment Anything Model (MedSAM) and analyze the segmented object using radiomics.

In this example, you create an interactive setup that employs MedSAM to segment an object from the specified ROI, and then analyzes the segmented object using radiomics. To get started with MedSAM, see Get Started with Medical Segment Anything Model for Medical Image Segmentation. To get started with radiomics, see Get Started with Radiomics.

This example requires the Medical Imaging Toolbox™ Model for Medical Segment Anything Model. You can install the Medical Imaging Toolbox Model for Medical Segment Anything Model from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. The Medical Imaging Toolbox Model for Medical Segment Anything Model requires Deep Learning Toolbox™ and Computer Vision Toolbox™.

About the Data Set

This example uses a subset of the Sunnybrook Cardiac Data data set [1][2]. The subset consists of 45 cine-MRI images and their corresponding ground truth label images. The MRI images were acquired from multiple patients with various cardiac pathologies. The MRI images are in the DICOM file format. The total size of the subset of data is ~105 MB.

Download Data Set

Download and unzip the data set.

zipFile = matlab.internal.examples.downloadSupportFile("medical","CardiacMRI.zip");
filepath = fileparts(zipFile);
unzip(zipFile,filepath)

Set the path imageDir to the folder that contains the downloaded and unzipped data set.

imageDir = fullfile(filepath,"Cardiac MRI");

Load a cine-MRI image from the data set into the workspace. Visualize the image.

medImg = medicalImage(fullfile(imageDir,"images","SC-HF-I-01","SC-HF-I-01_rawdcm_099.dcm"));
img = medImg.Pixels;
figure
imshow(img,[])

Extract Image Embeddings Using MedSAM

Load a MedSAM by creating a medicalSegmentAnythingModel object.

medsam = medicalSegmentAnythingModel;

Extract the image embeddings of the MRI image from the image encoder of the model using the extractEmbeddings object function.

embeddings = extractEmbeddings(medsam,img);

Configure Interactive Interface

Prepare the image display for MedSAM interactive segmentation. Create a new figure with a maximized window and axes, and display the image.

f = figure;
ax = axes(f);
dispIm = imshow(img,[],Parent=ax);
f.WindowState="maximized";

Initialize a text variable t to use to add analysis details later.

t = text(ax,0,0," ",Color="yellow");

Draw an ROI that contains an object to segment as a Rectangle object on the image by using the drawrectangle function.

roi = drawrectangle(ax);

Interactively Segment Object in Medical Image

To segment the object, run the MedSAM decoder on the image embeddings using the segmentObjectsFromEmbeddings object function. Specify the bounding box visual prompt BoundingBox as the position of the rectangle ROI.

mask = segmentObjectsFromEmbeddings(medsam,embeddings,size(img),BoundingBox=roi.Position);
overlayedImg = insertObjectMask(rescale(img),mask);

Visualize the segmentation mask overlaid on the image in the image display.

dispIm.CData = overlayedImg;

Analyze the segmented object using the analyzeObject helper function provided at the end of this example. The analyzeObject helper function performs radiomics analysis in the region indicated by the segmentation mask returned by the MedSAM decoder, and extracts key features such as surface area, major axis length, minor axis length, mean intensity, and intensity variance of the segmented region. The function then displays these key radiomics features as text near the segmented region in the image.

analyzeObject(img,mask,t,roi.Position);

Interactively Segment Object in Movable ROI

To automatically segment an object every time you move or resize an ROI around it on the image display, configure a listener for the moving ROI event by using the addlistener function. Specify the listener callback using the segmentAndAnalyzeROI helper function provided at the end of this example. The segmentAndAnalyzeROI helper function combines the steps for segmenting the ROI, visualizing the segmented mask, and analyzing the object into one function.

addlistener(roi,"ROIMoved",@(src,evt) ...
    segmentAndAnalyzeROI(evt.CurrentPosition,medsam,embeddings,img,dispIm,t));

Supporting Functions

analyzeObject

The analyzeObject helper function performs radiomics analysis in the region indicated by the segmentation mask returned by the MedSAM decoder, and extracts key features such as surface area, major axis length, minor axis length, mean intensity, and intensity variance of the segmented region. The function then displays these key radiomics features as text near the segmented region in the image.

function analyzeObject(img,mask,t,boxPrompt)
    imgVol = cat(3,img,zeros([size(img) 2],class(img)));
    volGeometry = medicalref3d(size(imgVol));
    data = medicalVolume(imgVol,volGeometry);
    mask = im2uint8(mask);
    maskVol = cat(3,mask,zeros([size(mask) 2],class(mask)));
    volGeometry = medicalref3d(size(maskVol));
    roiData = medicalVolume(maskVol,volGeometry);

    R = radiomics(data,roiData);
    S = shapeFeatures(R,Type="basic",SubType="2D");
    I = intensityFeatures(R,Type="IntensityBasedStatistics",SubType="2D");

    boxX = boxPrompt(1);
    boxY = boxPrompt(2);
    boxWidth = boxPrompt(3);
    boxHeight = boxPrompt(4);
    txtX = boxX + boxWidth + 10;
    txtY = boxY + boxHeight + 10;

    t.Position = [txtX txtY];
    t.String = ["Surface Area = " + S.SurfaceAreaMesh2D;
            "Major Axis Length = " + S.MajorAxisLength2D;
            "Minor Axis Length = " + S.MinorAxisLength2D;
            "Mean Intensity = " + I.MeanIntensity2D;
            "Intensity Variance = " + I.IntensityVariance2D];
end

segmentAndAnalyzeROI

The segmentAndAnalyzeROI helper function combines the steps for segmenting the ROI, visualizing the segmented mask, and analyzing the object into one function.

function segmentAndAnalyzeROI(boxPrompt,medsam,embeddings,img,dispIm,t)
    mask = segmentObjectsFromEmbeddings(medsam,embeddings,size(img),BoundingBox=boxPrompt);
    overlayedImg = insertObjectMask(rescale(img),mask);
    dispIm.CData = overlayedImg;
    analyzeObject(img,mask,t,boxPrompt)
end

References

[1] Radau, Perry, Yingli Lu, Kim Connelly, Gideon Paul, Alexander J Dick, and Graham A Wright. “Evaluation Framework for Algorithms Segmenting Short Axis Cardiac MRI.” The MIDAS Journal, July 9, 2009. https://doi.org/10.54294/g80ruo.

[2] “Sunnybrook Cardiac Data – Cardiac Atlas Project.” Accessed January 10, 2023. http://www.cardiacatlas.org/studies/sunnybrook-cardiac-data/.