extractEmbeddings

Extract image embeddings from Medical Segment Anything Model (MedSAM) encoder

Since R2024b

Syntax

embeddings = extractEmbeddings(medsam,I)

Description

embeddings = extractEmbeddings(medsam,I) extracts the embeddings of the input medical image I from the encoder of a Medical Segment Anything Model (MedSAM), medsam, by running a forward pass on the image encoder.

Note

This functionality requires Deep Learning Toolbox™, Computer Vision Toolbox™, and the Medical Imaging Toolbox™ Model for Medical Segment Anything Model. You can install the Medical Imaging Toolbox Model for Medical Segment Anything Model from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

Examples

Extract Embeddings from Medical Image

Create a Medical Segment Anything Model (MedSAM) object.

medsam = medicalSegmentAnythingModel;

Load a heart ultrasound image into the workspace. Visualize the image.

img = imread("heartUltrasoundImage.png"); 
figure
imshow(img,[])

Extract the image embeddings from the heart ultrasound image.

embeddings = extractEmbeddings(medsam,img);

Visualize some of the embeddings, which capture the features of the image.

figure
montage(embeddings(:,:,1:6),DisplayRange=[])

Input Arguments

medsam — Medical Segment Anything Model

medicalSegmentAnythingModel object

Medical Segment Anything Model, specified as a medicalSegmentAnythingModel object.

I — Image or batch of images

2-D numeric matrix | 3-D numeric array | 4-D numeric array

Image or batch of images from which to extract embeddings, specified as a 2-D numeric matrix, 3-D numeric array, or 4-D numeric array depending on the type and number of images.

Type of Image	Number of Images	Data Format
Grayscale image	Single 2-D image	2-D numeric matrix of size H-by-W
Grayscale image	Batch of B 2-D images	4-D numeric array of size H-by-W-by-1-by-B
RGB image	Single 2-D image	3-D numeric array of size H-by-W-by-3
RGB image	Batch of B 2-D images	4-D numeric array of size H-by-W-by-3-by-B

H and W are the height and width, respectively, of the input image I. To run the segmentation on a GPU, specify the input image I explicitly as a gpuArray (Parallel Computing Toolbox).

Data Types: single | double | int8 | int16 | int32 | uint8 | uint16 | uint32

Output Arguments

embeddings — Image embeddings

64-by-64-by-256 numeric array | 64-by-64-by-256-by-B numeric array

Image embeddings extracted from the Medical Segment Anything Model encoder, returned as a 64-by-64-by-256 or 64-by-64-by-256-by-B numeric array, depending on the type and number of input images.

Number of Images	Embeddings Format
Single grayscale or RGB image	64-by-64-by-256 array
Batch of B grayscale or RGB images	64-by-64-by-256-by-B array

The image embeddings are representations of corresponding input images that encode the spatial information, features, and context of the input images.

Extended Capabilities

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

This function fully supports GPU arrays. For more information, see Image Processing on a GPU.

Version History

Introduced in R2024b