object segmentation using CNN deep learning

Hi,

I'm trying to segment a specific object from an image using convolutional neural network (CNN).

I have a 2 part question:

1. I trained an image segmetation network using CNN - that mange to DETECT if this object is in the image or not. But, is there a way to "extract" the list of pixels that the network found to be related to the object? I need it in order to "crop" all the backgroung and leave only the object I'm intrested with.

In case there is no way to do it, and need to train a "Semantic Segmentation Network" for lable each pixel in image:

2. I'm not sure what supposed to be in the path that "labelDir" leads to (in the function: pixelLabelDatastore below). Image files that store pixel label data? if so - how

do I extract that? I have only PNG Images to train/validate with...this is not so clear to me.

Thanks in advance!

Answer 1

[Garmit Pant]

Hello YA

As per my understanding, the task that you need to achieve is Semantic Segmentation of Objects in images. To address your queries:

1) Objects can be detected and localised in an image using object detectors. MATLAB offers a range of object detectors that will help you detect and locate the object in image by returning bounding boxes and confidence scores. You can refer to the following example that uses YOLO v4 to detect objects.

https://www.mathworks.com/help/vision/ug/object-detection-using-yolov4-deep-learning.html

To separate the object from the background, you need to train and use a Semantic Segmentation Network. MATLAB supports multiple deep learning networks for semantic segmentation. The following example employs Deeplab v3+ for semantic segmentation:

https://www.mathworks.com/help/vision/ug/semantic-segmentation-using-deep-learning.html

2) For training semantic segmentation networks, you need pixel-wise label data where each pixel value corresponds to the class of that pixel. For example, if you have two classes (background and object), you might use 0 for background and 1 for the object. You can use MATLAB’s Image Labeler App. Kindly follow the guide below to understand the pixel labelling workflow:

https://www.mathworks.com/help/vision/ug/label-pixels-for-semantic-segmentation.html

After creating and saving the labels in a library named ‘labelDir’, you can create a pixel label datastore to use for training the semantic segmentation network. The following code snippet demonstrates a general workflow to create a training datastore to use for training semantic segmentation networks:

% Define the class names and their corresponding label IDs 
classNames = ["background", "object"]; 
labelIDs = [0, 1]; % Assuming 0 for background and 1 for object 

% Create a pixelLabelDatastore 
pxds = pixelLabelDatastore(labelDir, classNames, labelIDs); 

% Create an imageDatastore for your images 
imds = imageDatastore(imageDir); 

% Combine the imageDatastore and pixelLabelDatastore 
trainingData = combine(imds, pxds); 

The above steps and guides will help you perform semantic segmentation in MATLAB.

I hope you find the above explanation and suggestions useful!