"GPU"
"Invalid training data. Predictors and responses must have the same number of observations"
40 views (last 30 days)
Show older comments
massoud pourmandi
on 30 Jun 2022
Commented: Image Analyst
on 6 Oct 2023
I'd be appreciated it if someone help me to debug my code. Thanks in advance.
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec)
y = (1-x).^(node_degs_vec-1);
end
Error using trainNetwork
Invalid training data. Predictors and responses must have the same number of observations.
Error in DAGNET (line 49)
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
Accepted Answer
Cris LaPierre
on 1 Jul 2022
Note that, in the image you shared, they also all collapse back to a single node. My suspicion, then, is that you would use the lambdas to weight the result of each value. Again, you would know more about the particulars here than me.
I defined a vector of lambdas and used pagemtimes to compute the output of nodeDegreeLayer
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
lambda = [0.1 0.3 0.2 0.15 0.25]';
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
% Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec,lambda),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec,lambda),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec,lambda)
y = pagemtimes((1-x).^(node_degs_vec-1),lambda);
end
2 Comments
l
on 6 Oct 2023
Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:
gwsdata = reshape(gwsdata,205,3627);
tempdata = reshape(tempdata,205,3627);
ndvidata = reshape(ndvidata,205,3627);
precdata = reshape(precdata,205,3627);
petdata = reshape(petdata,205,3627);
lstdata = reshape(lstdata,205,3627);
smdata = reshape(smdata,205,3627);
%划分训练集,验证集
[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集,20%的数据作为测试集
temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);
ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);
prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);
pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);
lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);
sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);
gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);
temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);
ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);
prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);
pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);
lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);
sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);
gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);
%数据归一化
temp_train_norm = normalize(temp_train);
temp_test_norm = normalize(temp_test);
%ndvi_train_norm = normalize(ndvi_train);
%ndvi_test_norm = normalize(ndvi_test);
prec_train_norm = normalize(prec_train);
prec_test_norm = normalize(prec_test);
pet_train_norm = normalize(pet_train);
pet_test_norm = normalize(pet_test);
lst_train_norm = normalize(lst_train);
lst_test_norm = normalize(lst_test);
sm_train_norm = normalize(sm_train);
sm_test_norm = normalize(sm_test);
%gws_train_norm = normalize(gws_train);
%gws_test_norm = normalize(gws_test);
% 构造网络结构
layers = [
imageInputLayer([39 93 6],"Name","imageinput")
convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])
reluLayer("Name","Relu_1")
maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_1")
convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])
reluLayer("Name","Relu_2")
maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_2")
convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])
reluLayer("Name","Relu_3")
maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])
dropoutLayer(0.3,"Name","dropout_3")
convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])
reluLayer("Name","Relu_4")
fullyConnectedLayer(200,"Name","fc_1")
reluLayer("Name","relu_5")
dropoutLayer(0.4,"Name","dropout_4")
fullyConnectedLayer(1164*2783,"Name","fc_2")
regressionLayer("Name","Output")];
lgraph = layerGraph(layers);
options = trainingOptions('adam', ...
'MaxEpochs', 32, ...
'MiniBatchSize', 15, ...
'Shuffle', 'every-epoch', ...
'InitialLearnRate', 1e-2, ...
'LearnRateDropFactor', 0.01, ...
'LearnRateDropPeriod', 10, ...
'LearnRateSchedule','piecewise',...
'Plots', 'training-progress',...
'ExecutionEnvironment', 'cpu', ...
"L2Regularization", 0.001,...
'Verbose', true);
net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);
Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.
But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!
Cris LaPierre
on 6 Oct 2023
Without your data, it is pretty hard to provide a specific answer. However, if you have the same error, then your predictor and response inputs do not have the same number of observations.
More Answers (3)
Image Analyst
on 1 Jul 2022
Instead of
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
maybe
Xtemp=decoding_threshold*rand([sample_size, 1]); % Column vector. But why random numbers?!?
Xtrain=Xtemp;
Ytrain=zeros(size(Xtrain)); % Huh?!?
but it looks like you're training on random inputs, and a desired ground truth of all zeros. What's up with that? That makes no sense, at least not to me.
3 Comments
l
on 6 Oct 2023
Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:
gwsdata = reshape(gwsdata,205,3627);
tempdata = reshape(tempdata,205,3627);
ndvidata = reshape(ndvidata,205,3627);
precdata = reshape(precdata,205,3627);
petdata = reshape(petdata,205,3627);
lstdata = reshape(lstdata,205,3627);
smdata = reshape(smdata,205,3627);
%划分训练集,验证集
[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集,20%的数据作为测试集
temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);
ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);
prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);
pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);
lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);
sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);
gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);
temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);
ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);
prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);
pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);
lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);
sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);
gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);
%数据归一化
temp_train_norm = normalize(temp_train);
temp_test_norm = normalize(temp_test);
%ndvi_train_norm = normalize(ndvi_train);
%ndvi_test_norm = normalize(ndvi_test);
prec_train_norm = normalize(prec_train);
prec_test_norm = normalize(prec_test);
pet_train_norm = normalize(pet_train);
pet_test_norm = normalize(pet_test);
lst_train_norm = normalize(lst_train);
lst_test_norm = normalize(lst_test);
sm_train_norm = normalize(sm_train);
sm_test_norm = normalize(sm_test);
%gws_train_norm = normalize(gws_train);
%gws_test_norm = normalize(gws_test);
% 构造网络结构
layers = [
imageInputLayer([39 93 6],"Name","imageinput")
convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])
reluLayer("Name","Relu_1")
maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_1")
convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])
reluLayer("Name","Relu_2")
maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_2")
convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])
reluLayer("Name","Relu_3")
maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])
dropoutLayer(0.3,"Name","dropout_3")
convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])
reluLayer("Name","Relu_4")
fullyConnectedLayer(200,"Name","fc_1")
reluLayer("Name","relu_5")
dropoutLayer(0.4,"Name","dropout_4")
fullyConnectedLayer(1164*2783,"Name","fc_2")
regressionLayer("Name","Output")];
lgraph = layerGraph(layers);
options = trainingOptions('adam', ...
'MaxEpochs', 32, ...
'MiniBatchSize', 15, ...
'Shuffle', 'every-epoch', ...
'InitialLearnRate', 1e-2, ...
'LearnRateDropFactor', 0.01, ...
'LearnRateDropPeriod', 10, ...
'LearnRateSchedule','piecewise',...
'Plots', 'training-progress',...
'ExecutionEnvironment', 'cpu', ...
"L2Regularization", 0.001,...
'Verbose', true);
net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);
Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.
But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!
Image Analyst
on 6 Oct 2023
Start your own question and put all these variables (temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options) into a .mat file and attach it with the paperclip icon. Also give your code there in your new question.
Cris LaPierre
on 30 Jun 2022
I believe the problem is due to the fact that Xtrain is a cell array. I would replace your for loop with the following code instead:
Xtrain=Xtemp;
However, that will lead to a second error:
Error using trainNetwork
Targets and network outputs must be the same size. The networks outputs have size 1×5×1000 and the targets have size 1×1×1000.
This is because the network output (result of nodeDegree) is a 1000 x 5. You need to apply your knowledge is this space to have the network output and target variable be the same size. Here is a configuration that got the training to run for me.
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
% Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec)
y = sum((1-x).^(node_degs_vec-1),2);
end
0 Comments
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!