Why using fitnet is not giving 0.005 error with ENGINE Data set?

I am running the nftool in order to train a neural net using the example data "ENGINE" data set, I am using the same approach used here: mathworks.com/help/nnet/ug/choose-a-multilayer-neural-network-training-function.html#bss4gz0-26 for this data set, but I can not get an MSE less than 300 (and 0.005 is expected). I am using "trainlm" function, I am using a 2-30-2 architecture with "tansig" transfer function, I even tried with 2-50-2, but I can't get a good MSE (I have retrained maaany times). I assume that in the web page they used a simple "fitnet" with the conditions given. Is there something else needed to solve this specific example with an error equal or less to 0.005? I am using MATLAB R2012a.

If you can provide the script with the problem solved I would really appreciate it. Please help!

Thank you

Answer 1

[Greg Heath]

I generally consider a design successful if it can account for 99% of the mean target variance. The corresponding R^2 (Google Wikipedia R-squared) and normalized mean-square-error, NMSE = 1-R^2 are 0.99 and 0.01, respectively. NMSE = MSE/MSE00 where MSE00 = mean(var(t',1)) is the MSE of the naive constant output model that, regardless of input, yields the output y00 = repmat(mean(t,2),1,N). A goal of NMSE <= 0.005 = 5e-3 is reasonable. Given the target scale calculated below, a goal of MSE <= 5e-3 is DEFINITELY not.

 close all, clear all, clc
 [ x, t ] = engine_dataset;
 [I N ]   = size(x)  % [ 2 1199 ]
 [O N ]   = size(t)  % [ 2 1199 ]
 minmaxx  = minmax(x)
 minmaxt  = minmax(t)
% minmaxx =  1e3 *[ 0.0006 0.3140 
%                   0.5762 1.8018 ]
% minmaxt =  1e3 *[ -0.1767   1.7843
%                       0     1.7740 ]
  MSE00 = mean(var(t',1))     % 1e5*2.5894
  net   = fitnet;             % default H=10
  rng(0)
  [net tr y e ] = train(net,x,t);
  MSE           = mse(e)         % 1960.2
  NMSE          = MSE/MSE00      % 7.6e-3

Which, is smaller than my choice of 1e-2 but larger than 5e-3. The latter goal probably can be achieved with another state of the RNG; if not, one could resort to increasing H.

With the default 0.7/0/15/0/15 data division, the number of training examples and corresponding training equations are

 Ntrn = N-2*round(0.15*N)  % 839
 Ntrneq = Ntrn*O           % 1678

With a I-H-O node topology, the number of unknown weights is

 Nw = (I+1)*H+(H+1)*O

Minimum MSE solutions tend to be very robust w.r.t. noise, interference and measurement error, when Ntrneq >> Nw or equivalently H << Hub where

 Hub = -1+ceil( Ntrneq - O) / ( I + O 1))   % 335

Therefore, given the amount of data, a robust solution with H=30 is a reasonable goal. However, MSE <= 5e-3 is not.

Hope this helps

Thank you for formally accepting my answer

Greg

[Miguel]

Thank you very much Greg, that is somehow what I thought.

But I am wondering about something else: if you go this link:

http://www.mathworks.com/help/nnet/ug/choose-a-multilayer-neural-network-training-function.html#bss4gz0-26

And go to the ENGINE part, you'll see the plot of the mean squared error vs time, and you'll see that it goes to 5e-3; so how did they achieve that then? my first thought is that they normalized the target matrix somehow, am I correct?

[Miguel]

Or they just graphed the NMSE instead of the MSE?

[farzad]

The

MSE           = mse(e)         % 1960.2
  NMSE          = MSE/MSE00      % 7.6e-3

only works in case the number of Inputs = Number of Outputs otherwise , there will be a problem in size of the matrices

[Greg Heath]

Both MSE00 and MSE are scalars

Answer 2

[Greg Heath]

I don't have the slightest idea. It looks like the result of normalization. Contact MATLAB and find out who did it.

[Miguel]

Thank you very much anyway.