mutiple-input deep learning does badly in the validation set,it does well in single-input

My image data is captured from multiple sensors at three resolutions.Idesign a three-input deep net work which does badly in the validation set.But single-input that is the same net as three-input all do good.Can you give me some advice to improve my three-input net.

This is my three-input net.

numHiddenDimension=20; % speficy the dimension of the hidden layer
layers1 = createSimpleLayer(XTrain1,numHiddenDimension);
layers2 = createSimpleLayer(XTrain2,numHiddenDimension);
layers3 = createSimpleLayer(XTrain3,numHiddenDimension);

layers2=renameLayer(layers2,'_2');
layers3=renameLayer(layers3,'_3');

layersAdd=[fullyConnectedLayer(20,'Name','fcAdd1')
    fullyConnectedLayer(numClasses,'Name','fcAdd2')];

layersRemoved=[layers1(1:end);concatenationLayer(1,3,'Name','cat');layersAdd];

lgraphAggregated = addLayers(layerGraph(layersRemoved),layers2(1:end));
lgraphAggregated = addLayers(lgraphAggregated ,layers3(1:end));


lgraphAggregated = connectLayers(lgraphAggregated,'fc_2','cat/in2');
lgraphAggregated = connectLayers(lgraphAggregated,'fc_3','cat/in3');
analyzeNetwork(lgraphAggregated);

function layers=createSimpleLayer(XTrainData_4D,numHiddenDimension)
    layers = [
        imageInputLayer([224 224 3],"Name","imageinput","Mean",mean(XTrainData_4D,4))
        convolution2dLayer([7 7],96,"Name","conv_1","Padding","same")
        reluLayer("Name","relu_1")
        maxPooling2dLayer([2 2],"Name","maxpool_1","Stride",[2 2])
        convolution2dLayer([5 5],32,"Name","conv_2","Padding","same")
        reluLayer("Name","relu_2")
        maxPooling2dLayer([2 2],"Name","maxpool_2","Stride",[2 2])
        convolution2dLayer([3 3],16,"Name","conv_3","Padding","same")
        reluLayer("Name","relu_3")
        maxPooling2dLayer([2 2],"Name","maxpool_3","Stride",[2 2])
        convolution2dLayer([3 3],8,"Name","conv_4","Padding","same")
        reluLayer("Name","relu_4")
        fullyConnectedLayer(numHiddenDimension,"Name","fc")];
end

This is my figure of three- input.

This is the result.

The net of single net; the result

[yanqi liu]

yes，sir，if possible，may be upload some image dataset to analysis

Answer 1

[Pratyush Swain]

Hi neal,

I understand your three input model is currently not performing well as compare to single input model.Here are some suggestions to improve the performance of your three-input network:

1-Model Complexity: The three-input network is more complex than the single-input networks. If the dataset is not large enough, the model might not be able to learn effectively. Consider simplifying the model or collecting more data.

2-Architecture Tweaks: Adjust the architecture of the network for each input stream. If the data from different sensors have different characteristics, they might benefit from different types of layers or architectures.

3-Feature Fusion: The point at which you fuse the features from the different inputs can have a significant impact on performance. Experiment with different fusion strategies, such as fusing features earlier or later in the network, or using different methods.

4-Data Normalization: Ensure that the input data from all three sensors is normalized in the same way. Differences in the scale and distribution of the data can lead to poor performance when the inputs are combined.

5-Data Augmentation: Use data augmentation to increase the diversity of your training set, which can help improve the generalization ability of the network.

6-Ensemble Methods: Instead of using a single multi-input network, you could train separate networks for each input and then combine their predictions in an ensemble method. This could involve voting, averaging or methods of combining outputs.

The above techniques should help improve the performance of three input model. For more information,please refer to https://www.mathworks.com/help/deeplearning/ug/multiple-input-and-multiple-output-networks.html

Hope this helps.