"Invalid training data. Predictors and responses must have the same number of observations"

40 views (last 30 days)
I'd be appreciated it if someone help me to debug my code. Thanks in advance.
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
Error using trainNetwork
Invalid training data. Predictors and responses must have the same number of observations.
function y = nodeDegreeLayer(x, node_degs_vec)
y = (1-x).^(node_degs_vec-1);
end
Error using trainNetwork
Invalid training data. Predictors and responses must have the same number of observations.
Error in DAGNET (line 49)
net = trainNetwork(Xtrain,Ytrain,lgraph,options);

Accepted Answer

Cris LaPierre
Cris LaPierre on 1 Jul 2022
Note that, in the image you shared, they also all collapse back to a single node. My suspicion, then, is that you would use the lambdas to weight the result of each value. Again, you would know more about the particulars here than me.
I defined a vector of lambdas and used pagemtimes to compute the output of nodeDegreeLayer
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
lambda = [0.1 0.3 0.2 0.15 0.25]';
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
% Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec,lambda),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec,lambda),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec,lambda)
y = pagemtimes((1-x).^(node_degs_vec-1),lambda);
end
  2 Comments
l
l on 6 Oct 2023
Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:
gwsdata = reshape(gwsdata,205,3627);
tempdata = reshape(tempdata,205,3627);
ndvidata = reshape(ndvidata,205,3627);
precdata = reshape(precdata,205,3627);
petdata = reshape(petdata,205,3627);
lstdata = reshape(lstdata,205,3627);
smdata = reshape(smdata,205,3627);
%划分训练集,验证集
[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集,20%的数据作为测试集
temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);
ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);
prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);
pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);
lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);
sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);
gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);
temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);
ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);
prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);
pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);
lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);
sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);
gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);
%数据归一化
temp_train_norm = normalize(temp_train);
temp_test_norm = normalize(temp_test);
%ndvi_train_norm = normalize(ndvi_train);
%ndvi_test_norm = normalize(ndvi_test);
prec_train_norm = normalize(prec_train);
prec_test_norm = normalize(prec_test);
pet_train_norm = normalize(pet_train);
pet_test_norm = normalize(pet_test);
lst_train_norm = normalize(lst_train);
lst_test_norm = normalize(lst_test);
sm_train_norm = normalize(sm_train);
sm_test_norm = normalize(sm_test);
%gws_train_norm = normalize(gws_train);
%gws_test_norm = normalize(gws_test);
% 构造网络结构
layers = [
imageInputLayer([39 93 6],"Name","imageinput")
convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])
reluLayer("Name","Relu_1")
maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_1")
convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])
reluLayer("Name","Relu_2")
maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_2")
convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])
reluLayer("Name","Relu_3")
maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])
dropoutLayer(0.3,"Name","dropout_3")
convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])
reluLayer("Name","Relu_4")
fullyConnectedLayer(200,"Name","fc_1")
reluLayer("Name","relu_5")
dropoutLayer(0.4,"Name","dropout_4")
fullyConnectedLayer(1164*2783,"Name","fc_2")
regressionLayer("Name","Output")];
lgraph = layerGraph(layers);
options = trainingOptions('adam', ...
'MaxEpochs', 32, ...
'MiniBatchSize', 15, ...
'Shuffle', 'every-epoch', ...
'InitialLearnRate', 1e-2, ...
'LearnRateDropFactor', 0.01, ...
'LearnRateDropPeriod', 10, ...
'LearnRateSchedule','piecewise',...
'Plots', 'training-progress',...
'ExecutionEnvironment', 'cpu', ...
"L2Regularization", 0.001,...
'Verbose', true);
net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);
Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.
But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!
Cris LaPierre
Cris LaPierre on 6 Oct 2023
Without your data, it is pretty hard to provide a specific answer. However, if you have the same error, then your predictor and response inputs do not have the same number of observations.

Sign in to comment.

More Answers (3)

Image Analyst
Image Analyst on 1 Jul 2022
Instead of
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
maybe
Xtemp=decoding_threshold*rand([sample_size, 1]); % Column vector. But why random numbers?!?
Xtrain=Xtemp;
Ytrain=zeros(size(Xtrain)); % Huh?!?
but it looks like you're training on random inputs, and a desired ground truth of all zeros. What's up with that? That makes no sense, at least not to me.
  3 Comments
l
l on 6 Oct 2023
Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:
gwsdata = reshape(gwsdata,205,3627);
tempdata = reshape(tempdata,205,3627);
ndvidata = reshape(ndvidata,205,3627);
precdata = reshape(precdata,205,3627);
petdata = reshape(petdata,205,3627);
lstdata = reshape(lstdata,205,3627);
smdata = reshape(smdata,205,3627);
%划分训练集,验证集
[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集,20%的数据作为测试集
temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);
ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);
prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);
pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);
lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);
sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);
gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);
temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);
ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);
prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);
pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);
lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);
sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);
gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);
%数据归一化
temp_train_norm = normalize(temp_train);
temp_test_norm = normalize(temp_test);
%ndvi_train_norm = normalize(ndvi_train);
%ndvi_test_norm = normalize(ndvi_test);
prec_train_norm = normalize(prec_train);
prec_test_norm = normalize(prec_test);
pet_train_norm = normalize(pet_train);
pet_test_norm = normalize(pet_test);
lst_train_norm = normalize(lst_train);
lst_test_norm = normalize(lst_test);
sm_train_norm = normalize(sm_train);
sm_test_norm = normalize(sm_test);
%gws_train_norm = normalize(gws_train);
%gws_test_norm = normalize(gws_test);
% 构造网络结构
layers = [
imageInputLayer([39 93 6],"Name","imageinput")
convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])
reluLayer("Name","Relu_1")
maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_1")
convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])
reluLayer("Name","Relu_2")
maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_2")
convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])
reluLayer("Name","Relu_3")
maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])
dropoutLayer(0.3,"Name","dropout_3")
convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])
reluLayer("Name","Relu_4")
fullyConnectedLayer(200,"Name","fc_1")
reluLayer("Name","relu_5")
dropoutLayer(0.4,"Name","dropout_4")
fullyConnectedLayer(1164*2783,"Name","fc_2")
regressionLayer("Name","Output")];
lgraph = layerGraph(layers);
options = trainingOptions('adam', ...
'MaxEpochs', 32, ...
'MiniBatchSize', 15, ...
'Shuffle', 'every-epoch', ...
'InitialLearnRate', 1e-2, ...
'LearnRateDropFactor', 0.01, ...
'LearnRateDropPeriod', 10, ...
'LearnRateSchedule','piecewise',...
'Plots', 'training-progress',...
'ExecutionEnvironment', 'cpu', ...
"L2Regularization", 0.001,...
'Verbose', true);
net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);
Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.
But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!
Image Analyst
Image Analyst on 6 Oct 2023
Start your own question and put all these variables (temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options) into a .mat file and attach it with the paperclip icon. Also give your code there in your new question.

Sign in to comment.


Cris LaPierre
Cris LaPierre on 30 Jun 2022
I believe the problem is due to the fact that Xtrain is a cell array. I would replace your for loop with the following code instead:
Xtrain=Xtemp;
However, that will lead to a second error:
Error using trainNetwork
Targets and network outputs must be the same size. The networks outputs have size 1×5×1000 and the targets have size 1×1×1000.
This is because the network output (result of nodeDegree) is a 1000 x 5. You need to apply your knowledge is this space to have the network output and target variable be the same size. Here is a configuration that got the training to run for me.
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
% Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec)
y = sum((1-x).^(node_degs_vec-1),2);
end

massoud pourmandi
massoud pourmandi on 30 Jun 2022
  1 Comment
massoud pourmandi
massoud pourmandi on 30 Jun 2022
As you can see, functionlayer is responsible for expanding x value to a vector in that each element is related of x. Based on your modification, we have a scaler to scaler map not a scaler to vector map. I noticed that when we increrase the batch size to more than 1, It reflects itself in X. do you have any suguesstion to handle that?. thanks in advance.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!