How to train a deep learning network like a denoising network to learn a model using two image datasets when the images do not fit into memory?

How to train a deep learning network like a denoising network to learn a model using two image datasets when the images do not fit into memory?

I am trying to to build a network that can do a pixel wise estimation, i.e., if I train it with "before" and "after" images, the network can learn the manipulation on the image (image correction, denoise, object reorient, etc..). I want to learn a model such that, for example, if I have an image that has rain in the image, and then one without rain. Then the model should take away rain or, the model should predict the differences. I want to build a custom network like "denoisingNetwork".

I am using denoising as an example, but I am interested in other functionalities like image correction, object reorientation, etc.

How to specify the "before", and "after" image dataset to train the network when the images do not fit into memory?

Answer 1

[MathWorks Support Team]

In order to train a network like a denoising network when the images do not fit into memory, you need to feed the data by implementing a custom mini-batch datastore like the implementation of "associatedImageDatastore", which is a way to incorporate two datasets. Please refer to the following link for more information on the implementation of "associatedImageDatastore":

https://www.mathworks.com/help/deeplearning/ug/develop-custom-mini-batch-datastore.html#mw_1fbbfc62-d6e2-4e7c-843f-67b467135050

The denoising networks like "dnCNNLayers" has a set of noisy images and another set of denoised images. In such case, we assume a linear relation between them, like as follows:

noisy = denoised + noise

The input to the network is the noisy image and we train the network to predict the noise. Therefore, we need to use a regression layer with mean squared error (MSE), as follows:

|| outputOfTheNetwork(noisy) – (denoised - noisy) ||^2

Since the images do not fit into memory, you need to implement a  a custom mini-batch datastore like the implementation of "associatedImageDatastore". In this case, you need to slightly modify the "read" function of "associatedImageDatastore". The read function needs to do the subtraction of noisy from the denoised dataset as follows:

[…]
        function [data,info] = read(ds)           
            % Read one batch of data
            inputImageData = read(ds.InputImds);
            outputImageData = read(ds.OutputImds);
            outputImageData = outputImageData - inputImageData;
[…]

Also, please note that this implementation assumes that input is noisy and the output is denoised. 

Similarly, you need to manipulate the "read" function of "associatedImageDatastore", according to the functionality that is required (like image correction, object reorient, etc.).

Further, the sizes of input and output batch should be the same to avoid errors, i.e., the mini-batch size should be consistent for input and output.