Invalid training data. X and Y must have the same number of observations.

I have long ECG signals segmented into 300 points segments/heartbeats. I want to use CNN for feature extraction with a bidirectional LSTM layer for classification. I have the following network:

inputSize=[1 300 1]; %the heartbeat size
Layers=[
sequenceInputLayer(inputSize,'Normalization', 'zscore', 'Name','input'); 
sequenceFoldingLayer('Name','fold')

convolution2dLayer([1 7], 16,'stride',[1 1], 'padding','same','Name','conv1')
batchNormalizationLayer('Name','bn1')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool1')

convolution2dLayer([1 7], 32,'stride',[1 1], 'padding','same','Name','conv2')
batchNormalizationLayer('Name','bn2')
reluLayer('Name','relu1')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool2')

convolution2dLayer([1 5], 64,'stride',[1 1], 'padding','same','Name','conv3') 
batchNormalizationLayer('Name','bn3')
reluLayer('Name','relu2')

convolution2dLayer([1 5], 128,'stride',[1 1], 'padding','same','Name','conv4')
batchNormalizationLayer('Name','bn4')
reluLayer('Name','relu3')

convolution2dLayer([1 3], 256,'stride',[1 1], 'padding','same','Name','conv5')  
batchNormalizationLayer('Name','bn5')
reluLayer('Name','relu4')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool3')

convolution2dLayer([1 3], 512,'stride',[1 1], 'padding','same','Name','conv6')
batchNormalizationLayer('Name','bn6')
reluLayer('Name','relu5')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool4')

sequenceUnfoldingLayer('Name','unfold')
flattenLayer('Name','flatten')

bilstmLayer(200,'Name','lstm')

reluLayer('Name','relu6')
fullyConnectedLayer(256,'Name','fc1')
reluLayer('Name','relu7')
fullyConnectedLayer(128,'Name','fc2')
reluLayer('Name','relu8')
fullyConnectedLayer(5,'Name','fc3')
softmaxLayer('Name','softmax')
classificationLayer('Name','classification')
];

I connect the layers using:

lgraph = layerGraph(Layers);
lgraph = connectLayers(lgraph,'fold/miniBatchSize','unfold/miniBatchSize');

When I train the network I have the following error:

Error using trainNetwork (line 170)
Invalid training data. X and Y must have the same number of observations.

Error in CNN_LSTM (line 152)
convnet = trainNetwork(Xtrain,Ytrain,lgraph,options);
    

Xtrain has the size 1 300 1 91147 (it contains 91147 segments of 300 datapoints each)

Y train has the size 91147 1

Can someone please tell me how could I solve this error? I checked the documentation Sequence Classification Using Deep Learning but it did not help me.. If I introduce the S= sequence length (91147) I have another error

Answer 1

[Tomaso Cetto]

Hi Ioana,

There are a few modifications to your data and network that are required to make your workflow possible.

The difficulty is associated with the fact that the convolutional part of your network needs to operate on data of the form S x S x C x B (in your case, this is equal to 1 x L x 1 x 91147, where L = 300 to start with, as you rightly pointed out), but the recurrent part of your network (that is, the bilstmLayer) needs data in the format C x B x T (in your case, 1 x 91147 x L, where L is the length of the segment once it's gone through all the convolution layers).

The issue here is that your dimension which I've marked as L is at first a spatial dimension, and then a time dimension. To make this switch possible, we are going to need a custom layer. More on that later - for now, let's start by thinking about how you want to specify your training data.

The sequenceInputLayer requires data in cell array format, as Srivardhan pointed out.

For your training data, you're going to need one cell array for each observation (so 91147 cell arrays), and in each of these cell arrays, you'll need a S x S x C x T array (see this part of the trainNetwork documentation for more details). In your case, these arrays will be of size 1 x 300 x 1 x 1 - the singleton T dimension is irrelevant here, because we've marked the segment length as a spatial dimension, but it'll become relevant later, when we use it in our custom layer. The following line of code shows how I've created random data in this format - you're going to want to get your 1-by-300-by-1-by-91147 data array in this format too:

inputSize = [1 300 1 1];
numTrainSamples = 91147;
xtrain = arrayfun(@(x)rand(inputSize),1:numTrainSamples,'UniformOutput',false)';

For your responses, because each of the 91147 sequences has a single label, you can keep the 91147 x 1 vector of categorical responses.

Now, on to the network. The main changes you need to make are the following:

1) Instead of the flatten layer, you will need to create a custom layer which transforms your S x S x C x B data to C x B x T data. If you are unfamiliar with custom layers, all you need to do is save the code below into its own file called 'imageToSequenceCustomLayer.m' and include this file in the directory with your main network training file:

classdef imageToSequenceCustomLayer < nnet.layer.Layer & nnet.layer.Formattable  
    methods
        function this = imageToSequenceCustomLayer(nanvargsme)
            arguments
                nvargs.Name (1,:) {mustBeText} = "imageToSequence"
            end
            this.Name = nvargs.Name;
        end
        
        function Y = predict(~, X)
            % Permute from S x S x C x B to C x B x T x S
            Y = permute(stripdims(X), [3 4 2 1]);
            
            % Re-label the array with the correct dimensions
            Y = dlarray(Y, 'CBT');
        end
    end  
end

You can then just replace your flattenLayer with

imageToSequenceCustomLayer()

The flattening is somewhat implicit in the permutation of the data that happens in the customLayer, so we don't need it anymore!

The final thing you'll need to do is to specify the 'OutputMode' NVP of your biltsmLayer as 'last', so that you don't classify every single time step of the sequence. Instead, you output the last time step and use that to classify the entire sequence.

Your network should look as shown below:

layers = [
    sequenceInputLayer([1 300 1],'Normalization', 'zscore', 'Name','input'); 

    sequenceFoldingLayer('Name','fold')

    convolution2dLayer([1 7], 16,'stride',[1 1], 'padding','same','Name','conv1')
    batchNormalizationLayer('Name','bn1')
    maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool1')

    convolution2dLayer([1 7], 32,'stride',[1 1], 'padding','same','Name','conv2')
    batchNormalizationLayer('Name','bn2')
    reluLayer('Name','relu1')
    maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool2')

    convolution2dLayer([1 5], 64,'stride',[1 1], 'padding','same','Name','conv3') 
    batchNormalizationLayer('Name','bn3')
    reluLayer('Name','relu2')
    
    convolution2dLayer([1 5], 128,'stride',[1 1], 'padding','same','Name','conv4')
    batchNormalizationLayer('Name','bn4')
    reluLayer('Name','relu3')
    convolution2dLayer([1 3], 256,'stride',[1 1], 'padding','same','Name','conv5')  
    batchNormalizationLayer('Name','bn5')
    reluLayer('Name','relu4')
    maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool3')
    
    convolution2dLayer([1 3], 512,'stride',[1 1], 'padding','same','Name','conv6')
    batchNormalizationLayer('Name','bn6')
    reluLayer('Name','relu5')
    maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool4')

    sequenceUnfoldingLayer('Name','unfold')

    imageToSequenceCustomLayer()

    bilstmLayer(200,'Name','lstm', 'OutputMode', 'last')

    reluLayer('Name','relu6')
    fullyConnectedLayer(256,'Name','fc1')
    reluLayer('Name','relu7')
    fullyConnectedLayer(128,'Name','fc2')
    reluLayer('Name','relu8')
    fullyConnectedLayer(5,'Name','fc3')

    softmaxLayer('Name','softmax')
    classificationLayer('Name','classification')
];

This should not throw any errors!

Hope this helps!

Tomaso

[Ioana Cretu]

Dear @Tomaso Cetto,

Thank you so much! You have no idea how much I appreciate this answer! I printed it, read it and also read the documentation associated with it. It seems it all make sense now. However, when I try to apply the changes I have an error

Error using imageToSequenceCustomLayer
File: imageToSequenceCustomLayer.m Line: 5 Column: 17
Function argument definition error in imageToSequenceCustomLayer constructor. Arguments block and function line must contain the same
arguments in the same order, including ignored arguments.

When I verified the documentation Function Argument Validation it seems everything is as expected. I am not sure about nvargs. I did not find any documentation about it (Variable-Length Argument Lists). Can you please explain me what is supposed to do and why do you think it gives me an error? Does it declare the type of input that is expected as a name?

Kind regards,

Ioana

[Tomaso Cetto]

Hi Ioana!

That's no problem at all, happy to help! The reason you're seeing that error is that I've mistakenly included a typo in the custom layer (sorry about that!) - this is the constructor function I wrote up for the layer:

function this = imageToSequenceCustomLayer(nanvargsme)
    arguments
        nvargs.Name (1,:) {mustBeText} = "imageToSequence"
    end
    this.Name = nvargs.Name;
end

You'll see I have 'nanvargsme' as the input argument, when it should be 'nvargs' - not sure how that's happened! Changing it to 'nvargs' should fix it for you, but let me know if it doesn't.

Best,

Tomaso

[Ioana Cretu]

Hi again!

Thank you for your response

I changed it to:

classdef imageToSequenceCustomLayer < nnet.layer.Layer & nnet.layer.Formattable  
    methods
        function this = imageToSequenceCustomLayer(nvargs)
            arguments
                nvargs.Name (1,:) {mustBeText} = "imageToSequence"
            end
            this.Name = nvargs.Name;
        end
        
        function Y = predict(~, X)
            % Permute from S x S x C x B to C x B x T x S
            Y = permute(stripdims(X), [3 4 2 1]);
            
            % Re-label the array with the correct dimensions
            Y = dlarray(Y, 'CBT');
        end
    end  
end

Unfortunately now I have another error:

Error using imageToSequenceCustomLayer
The specified superclass 'nnet.layer.Formattable' contains a parse error, cannot be found on MATLAB's search path, or is shadowed by
another file with the same name.

Best,

Ioana

[Tomaso Cetto]

Hi Ioana,

Do you mind me asking which version of MATLAB you're running? The ability to change data dimensions in a custom layer through nnet.layer.Formattable was a new feature added in 2021a. Is that the release you're working with?

Best,

Tomaso

[Tomaso Cetto]

If you do find that you don't have access nnet.layer.Formattable, there should be a way to get this to work without it. The first modification needed is to slightly change the permute operation, as shown below:

classdef imageToSequenceCustomLayer < nnet.layer.Layer %& nnet.layer.Formattable
    
    methods
        function this = imageToSequenceCustomLayer(nvargs)
            arguments
                nvargs.Name (1,:) {mustBeText} = "imageToSequence"
            end
            this.Name = nvargs.Name;
        end
        
        function Y = predict(~, X)
            
            % Permute S1 x S2 x C x B x T to S1 x T x C x B x S2 - here S1
            % and T will both be singleton, and S2 is the dimension that we
            % care about (representing the signal length)
            Y = permute(X, [1 5 3 4 2]);
        end
    end  
end

The difference here is that, without nnet.layer.Formattable, I can't remove the spatial dimensions from the array, so I have to keep them there, even though they're singleton. The knock-on effect of this is that we need to re-introduce our flattenLayer to come after the custom layer - that will flatten the spatial dimensions for us, leaving us with a C x B x T the lstmLayer can accept!

Let me know if this works for you!

Best,

Tomaso

[Ioana Cretu]

Dear Tomaso,

Thank you again for all the help. I have 2020a version. Unfortunately the university does not have access to the new version yet. I made the modifications and added the flattenLayer after the custom one, but it gives me another error related to the name of the layer:

Undefined function 'mustBeText' for input arguments of type 'string'.

Error in imageToSequenceCustomLayer

Error in CNN_LSTM (line 119)
    imageToSequenceCustomLayer()   

The 'mustBeText' function exists in the documentation. It should work..

Kind regards,

Ioana

[Tomaso Cetto]

Hi Ioana,

Thanks for the information! Unfortunately, the fact you are working with 2020a precludes you from taking advantage of custom layers which permute a spatial dimension to a temporal dimension, and use trainNetwork. So even if you remove the 'mustBeText' validator from your custom layer (the reason it's erroring is because it's not supported in 2020a), you'll run into issues further along.

To get your workflow to function in the release you're working with, you'll have to move to a custom training loop with a dlnetwork. This way of building models give much more flexibility, and allows us to pass from the CNN part of your network to the LSTM part seamlessly.

The file I've attached shows how to convert your workflow from the trainNetwork API to the dlnetwork API. Because I don't have access to your data, I wasn't able to optimize it for accuracy, so you'll probably have to tune hyper-parameters as needed. The main things to be aware of now that we've moved over to a custom training loop are the following:

1) I've split the network into two parts - the CNN part and the LSTM part. The link between them (where we permute our data as it comes out the first network) happens in the modelGradients function. Because the first network is really just a standard 2D CNN, I've used an imageInputLayer for it. The second network also needs its input layer, which is going to be a sequenceInputLayer.

2) The consequence of using an imageInputlayer for the first network is that you can now just create an array containing all your data in S x S x C x B format (in your case, this will be 1 x 300 x 1 x 91147), and then just batch that in the custom training loop.

Let me know if this makes sense and if you run into any problems! A useful resource to get more information about custom training loops is the 'Train Network Using Custom Training Loop' doc example (which served as a basis for the script I've attached).

All the best,

Tomaso

[Ioana Cretu]

Dear Tomaso,

I am truly sorry for replying so late.

I understood what you told me in the comment, but I was a bit confused by the code, because you firt train the networks individually using the custom loop and then have the modelGradients function that links the CNN and LSTM in the prediction process, is that right? I saw in the documentation 'Train Network Using Custom Training Loop' they had a similar example. Is this a individual training of the networks and when predicting you use both of the networks?

I run the code and I have this error in the custom loop when I run the code:

Error using dlfeval (line 43)
Size of predictions and target values arguments must match.

Error in CNNLSTM20a (line 133)
        [gradientsCNN,gradientsLSTM,state,loss] = dlfeval(@modelGradients,...ts,...
            

All the best,

Ioana

[Tomaso Cetto]

Hi Ioana,

Regarding the workings of the custom training loop, everything that happens in the code I've attached is the training phase - I didn't attach any testing/prediction code. The modelGradients function effectively holds the forward and backward pass for each iteration (or batch of data). It does 4 things, in the following order:

1. Forward the data through the CNN part of the network.
2. Permute the output of step 1 so that it can be fed to the LSTM part of the network.
3. Forward the data through the LSTM part of the network.
4. Call dlgradient on both networks (lines 171-172), which is the backwards pass where the gradients get updated with respect to the loss.

So the networks are trained jointly because we update both sets of learnables, even if we have to break down the forward pass into two calls to 'forward'.

As for the error you're seeing, are you seeing it when you run the code I sent you without modifying it, or after you've modified it to include your data and not my dummy data? If the latter, then you just need to ensure that your XTrain and YTrain variables match the size/layout of the ones that I provided. The error is being thrown because the size of what comes out of your network doesn't match the size of your targets, and so the loss can't be calculated on line 168. Your targets should be a 91147 x 1 categorical vector when you define them at the top, and after line 120 they should be a one-hot encoded numClasses x miniBatchSize array (5 x 128 in this case).

Let me know if it works for you!

Best,

Tomaso

[Ioana Cretu]

Dear Tomaso,

Thank you. I highly appreaciate that you have the patience to explain and make me understand.

I modified it but used a part of the patients, so my data had different dimensions. The error was my fault.

I modified it and I got another error:

Index in position 4 exceeds array bounds (must not exceed 1).

Error in CNNLSTM20a (line 166)
    Xtrain = Xtrain(:,:,:,idx);

So I used the below code and converted the data:

Xtrain = reshape(Xtrain,[1, 300, 1, length(Xtrain)]);
Xvalidation=reshape(Xvalidation,[height, width, channels, length(Xvalidation)]);
Xtest = reshape(Xtest,[height, width, channels, length(Xtest)]);

The code worked and I obtained this:

The results are clearly not good, but I will have to play with the hyperparameters. My next question, and hope not to disturb you with more questions, is how do I use this code to make prediction?

Best regards,

Ioana

[Tomaso Cetto]

Hi Ioana!

No problem at all, glad to see it's training for you!

As far as using the network for prediction, the 'Train Network Using Custom Training Loop' example in the documentation will hopefully be able to point you in the right direction, in particular the 'Test Model' section.

You'll see that what needs to be done is to define a seperate modelGradients function, called 'modelPredictions', which follows similar steps to the original modelGradients function. Importantly for your use case, you want to be sure to modify the for loop inside the function (which loops over batches of data), so that, when passing data through the network, you do what your modelGradients does - that is, call predict on the first network, permute the data, and then call predict on the second network, to return the right dlYPred .

Hope this helps!

Tomaso

[Ioana Cretu]

Dear Tomaso,

Yes, it really helps. Thank you!

I also managed to install the new version of matlab on my personal pc (2021a), and I tried the first solution out of curiosity (the one suggested by you above) using the imageToSequenceCustomLayer and I got this error:

Error in CNN_LSTM (line 145)
convnet = trainNetwork(Xtrain,Ytrain,lgraph,options);

Caused by:
    Layer 'imageToSequence': Input size mismatch. Error using 'predict' in Layer imageToSequenceCustomLayer. The function threw an
    error and could not be executed.
    

I checked the dimensions and they seem fine. I attached the used files to this comment. Can you please tell me why it does not work?

Best regards,

Ioana

Answer 2

[ytzhak goussha]

hey,

You should check the dimentions of your features and targets of your training data.

These are the dimentions you need:

features/inputs:

{HxWxCxS}

targets:

{1xS}

H - height

W - width

C - channle

S - time point

I think that in your case, you need to transpose the targets into 1xS array instead of Sx1

[Ioana Cretu]

@ytzhak goussha thank you for your response. I am not sure about how I should reshape the signal. I tryed to transpose the targets but it did not solve my error.

Taking into account the comment made by by Joss Knigh at this question I reshaped the signals along the 4th dimension like that:

• Xtraining: 1-by-300-by-1-by-91147; where 300 is the length of each signal and 91147 are the number of signals
• Ytraining: 91147-by-1- which contains the labels for 5 classes of all the 91147 samples (these are categorical)

How do you think I should reorganise them?