When can the found Neural network model be accepted ?

I have a question concerning a neural network model. If we have these errors of a neural network model:

MSE for training set= 1.5654 e-6 MSE for validation set= 7.4452 e-7 MSE for test set= 9.6667 e-6

Based on these errors, do we have to accept this neural network model or not? (If you notice the error decreases in the validation set but increases in the test set)

Thanks in advance,

Answer 1

[Greg Heath]

Your performance numbers are scale dependent. They should be normalized by the values that result if the model was so NAIVE that it outputs a constant value, REGARDLESS of the input. The constant that minimizes MSE is just the mean target variance.

 vart = mean(var(t',1))

then, if e is the error

 e = t-y,

the normalized mean-square-error and resulting Rsquare are given by

     NMSE = mse(e)/vart

      Rsq = 1 - NMSE

Rsq is the fraction of the mean target variance that is modeled by the net (See Wikipedia Rsquare).

My training set design goal for regression and classification is

     NMSEtrn = MSEtrn/varttrn <= 0.01

(99% of the training target variance modeled) whereas for openloop feedback timeseries training, it is 0.005 with the crossed finger hope that closing the feedback loop doesn't raise it above 0.01.

Multiple random initial weight designs are ranked by NMSEval, the slightly biased NMSE for the validation set. Finally, the completely unbiased estimate for current and unseen nontraining data is given by NMSEtst.

Finally, a design is usually chosen w.r.t. NONTRAINING NMSE.

Now I usually design 10 nets for each value of H (number of hidden nodes) considered. My selection is based on the nontraining values Rsqval and Rsqtst (regardless of Rtrn) provided Nval and Ntrn are sufficiently large ( I prefer

 Nval = Ntst >= 30.

However, sometimes I'm forced to accept

 Nval + Ntst >= 30).

Hope this helps.

Thank you for formally accepting my answer

Greg

P.S. Many of my earlier posts use the notations MSE00 and MSE00a for biased and unbiased estimates of the average target variance

[Ismail Belhaj]

Thanks a lot Greg for your help. By the way, the MSE I put is a normalized one. Before using the data, I normalized the data between 0 and 1 and I used a sigmoid function as a transfer function. When computing the errors, I set a condition While MSE (normalized) > 10e-7 in the training set. For the validation and test sets, I worked with the condition of epochs found in the training set, which is: ( While epoch < Nbr of epochs found). It gave me those errors which are normalized for training, validation and test sets. Since they were so small I couldn't know if the model should be accepted or not, also the number of epochs was very high ( more than 20 000 epochs) because I used a very big set of data. Please if you have anything to add, I'll appreciate it and again thanks a lot for the help