Couldn't fit the data using NEURAL NETWORKS IN MATLAB (fitnet function)

Hi,

i have been trying the fit the data to a nonlinear model using neural networks in matlab. i have several sets of data. and my code is working fine for some data sets but not for all the data sets.

for some data sets i am able to fit with good regression coefficient.for some sets of data it is giving me a constant value of output (i.e: almost '0' regression coefficient).

this is the architecture of my neural network: Feedforward neural network with back propagation. no of hidden layers-1 no of neurons in a hidden layer -i am varying to see the result in each run

CAN ANYONE POINT OUT WHAT IS GOING WRONG IN MY CODE PLEASE???

      clear;
      %%to load data from excel file
      filename='dD.xlsx';
      x=xlsread(filename);
      p=x(:,2:12);
      t=x(:,1);
      inputs = p';
      targets = t';
      % rng(200,'v4');
      rng(0)
      %%Create a Fitting Network
      hiddenLayerSize = 5;
      net = fitnet(hiddenLayerSize);

      %%Set up Division of Data for Training, Validation, Testing
      % net.divideFcn = 'dividetrain';  % No validation or test data
      net.divideParam.trainRatio = 100/100;
      net.divideParam.valRatio = 0/100;
      net.divideParam.testRatio = 0/100;

      net.trainFcn = 'trainbr';
      % net = configure(net,ptrans,tn);
      net.layers{1}.transferFcn = 'logsig';
      net.layers{2}.transferFcn = 'purelin';
      %%Train the Network
      [net,tr] = train(net,inputs,targets);
      tr.best_epoch;
      effective_param = tr.gamk;
      effective_no_of_parameters = effective_param(length(effective_param));

      wt_IL=net.IW{1,1};
      wt_HL= net.LW{2,1};
      bias_IL=net.b{1};
      bias_HL=net.b{2};
      %%Test the Network
      outputs = net(inputs);
      errors = gsubtract(outputs,targets);
      performance = perform(net,targets,outputs)

Answer 1

[Greg Heath]

1. The best approach for us to help is to

a. Apply your code to the MATLAB data set that provides the best example of your problem

 help nndatasets
 doc nndatasets

b. Since you have 11 inputs and 1 output, consider one of the following

 abalone_dataset       8  -> 1 ,  4177  Abalone shell rings dataset.
 bodyfat_dataset      13  -> 1 ,   252   Body fat percentage dataset.
 chemical_dataset      8 -> 1  ,   498   Chemical sensor dataset.
 house_dataset        13 ->  1  ,  506   House value dataset

2. Record the initial RNG state so results can be duplicated

3. Obtain the normalized mean-square error and tabulate the corresponding Rsquare (see Wikipedia) resulting from a double loop approach to determining the number of hidden nodes (outer loop h = Hmin:dH:Hmax) and initial random weights ( inner loop i = 1:Ntrials)

   MSE00  = mean(var(target',1))  % Reference MSE (optimal for constant output)
   NMSE    = mse(target-output)/MSE00
   Rsquare = 1 - NMSE

4. Typically, my goal is to use

   i) numel(Hmin:dH:Hmax) = 10; Ntrials = 10
   ii) Minimize the number of hidden nodes subject to the constraint that the net models at least 99% of the mean target variance, i.e., Rsquare > 0.99.

5. You wrote that you have varied h and made multiple runs. However, you did not

 a. State the size of your data set
 b. Tabulate your results (Rsquare is preferred, but NMSE is ok)

6. I have posted scores of examples in both NEWSGROUP and ANSWERS. A useful search combination is

 greg fitnet Ntrials

 7. The no-overfitting condition Hmax << Hub in my posts is not necessary when regularization (trainbr and or msereg) is used.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Kishore]

I couldn't understand some of your suggestions

can you please explain a bit more about "Record the initial RNG state so results can be duplicated" how to do perform this operation??

As i have less data i am using my entire dataset as training set only.

i didn't use any pre processing commands in the above code. when i tried using some pre processing tools like MAPMINMAX and PROCESSPCA but results got varied (some results were good, some were bad).

can you please explain me about what are all the pre processing and post processing tools are included in the function FITNET..i am not able to get much information in MANUAL???

i attached the size, no of neurons i used and R_square coefficients i got for different data sets while running the same CODE

Thanks for the help

[Greg Heath]

% I couldn't understand some of your suggestions % % can you please explain a bit more about "Record the initial RNG state so % results can be duplicated" how to do perform this operation??

You have already done it by explicitly specifying rng(0)

% As i have less data i am using my entire dataset as training set only.

That is dangerous because then it is hard for you to convince your teacher, sponsor or client that you have an unbiased estimate of performance on new or unseen data.

If this is serious work, you need to go back and get unbiased estimates using a test set. I recommend using 10 fold cross-validation enough times to create a good R2 histogram and the best 30 or more designs can be used to obtain convincing summary R2 statistic estimates.

Since initial weights are random, you will get some unsatisfactory designs which you should include in the histogram but exclude from the summary statistics.

I used a slightly modified code on the bodyfat_data set for H = 16:-1:0

             logsigresult              tansigresult 
           16      0.78952           16      0.79693
           15      0.78939           15      0.78933
           14      0.78923           14      0.78934
           13      0.78924           13      0.79693
           12      0.78951           12      0.79694
           11      0.78927           11      0.79694
           10       0.7895           10      0.79694
            9      0.78926            9      0.79695
            8      0.78956            8      0.79695
            7      0.78957            7      0.79696
            6      0.78948            6      0.79697
            5      0.78934            5      0.79699
            4      0.78958            4      0.79709
            3      0.77613            3      0.77828
            2      0.76794            2      0.76788
            1      0.75763            1      0.75741
            0     -0.20245            0      0.74731

It looks like the default tansig with H = 4 is a reasonable choice.

% i didn't use any pre processing commands in the above code. when i tried % using some pre processing tools like MAPMINMAX and PROCESSPCA but results % got varied (some results were good, some were bad).

MAPMINMAX is the default. However, I prefer explicitly using ZSCORE before training to delete or modify outliers. However, since it is too painful to continually remove MAPMINMAX I just use both.

In other high dimensional problems I have found decent input reduction just using a linear model and STEPWISEFIT. The weaker the model, the better it tends to choose good inputs. Also, original inputs are kept instead of linear combinations.

% can you please explain me about what are all the pre processing and post % processing tools are included in the function FITNET..i am not able to % get much information in MANUAL???

When you type, before and after training, WITHOUT THE ENDING SEMICOLON

net = net

you will see how to get that and other net details. Similarly

tr = tr

will yield training details

% i attached the size, no of neurons i used and R_square coefficients i got % for different data sets while running the same CODE

OK, will look at it.

% Thanks for the help

Will be looking for my check in the mail (;>)

[Greg Heath]

UH-OH

Having Microsoft Office problems.

Please email a *.txt or *.m copy

Thanks

Greg

[Kishore]

thanks for the reply.

What would happen if we change the seed in the RNG function?? like if i change RNG(0) to RNG(200).

I read somewhere that we can use entire data set as training set according to BAYSIAN REGULARIZATION THEORY...so i am using TRAINBR function. and total dataset as training set

I tried using my code on the body_fat data set, i am able to fit the data upto 0.79 regression coefficient. where can i confirm my results on body_fat dataset??

thanks

[Greg Heath]

Changing the seed just changes the initial state of the RNG. Therefore you would get different initial weights and a different trn/val/tst split.

Therefore, it is essential for reproducibility of results.

I've been designing NN classifiers since the 1970s. Regardless of what you say or reference, you cannot find a sponsor or client who is willing to part with $$$ unless you can get good performance on nontraining data!

You can confirm your results by comparing with mine.

Greg

PS I used Hmax = 16 because that is when Nw = Ntrneq. As expected, there are no indications of the onset of overtraining. If you are curious, you might want to compare without regularization or validation stopping.

[Greg Heath]

Wait a minute. You wrote

 i am able to fit the data upto 0.79 regression coefficient

I think you meant the coefficient of determination Rsquare = 0.79

The regression coefficient is R = sqrt(Rsquare) ~ 0.89

[Kishore]

Thanks..

Yes Rsquare is 0.79

I have two final questions

how should we proceed when we have less data and want to fit the model using neural network??

how do we set the parameters like "net.trainParam.goal = 0.01*Ndof*MSEtrn00a/Ntrneq; % R2a >=0.99"..as specified by you in the thread?? http://in.mathworks.com/matlabcentral/newsreader/view_thread/333427

[Greg Heath]

1.When you don't have enough data for Ntrneq >> Nw You can use some or all of the following. It is important (for convincing $ponsor$ and client$) to make multiple designs so that you can calculate robust summary statistics from the best 30 or more acceptable designs.

 a. Simulate extra data by creating Gaussian distributions about each data point. It is not unusual to use simulated data for training and validation; then use the original data to get "unbiased" performance estimates.

 b. Cross validation or Bootstrapping

 c. Validation Stopping

 d. Regularization

2. Don't understand your question. Be specific

[Kishore]

http://in.mathworks.com/matlabcentral/newsreader/view_thread/333427

as per my understanding on this thread. you have changed some default parameters of the transfer function (in this case TRAINLM).

eg: net.trainParam.goal = 0.01*Ndof*MSEtrn00a/Ntrneq;

how should i change this value when i use different transfer function like TRAINBR?? what is the necessity of chanding default values to some specific values??

[Greg Heath]

This change stops training when the training error variance is less than 1% of the original target variance.

You can make it smaller (e.g., by a factor of 2 or 10) if you wish.

TRAINBR doesn't have a different transfer function.

[Kishore]

Thanks for all these valuable comments.