How can cross validate neural networks

"I read the previous question and answers but I don't understand well. if I have 18 samples with 2 hidden layer and in each run inputs and targets have 17 samples(one more out) and target p is the which is out,is it correct to... net = newfit(inputs,targets,2); [net,tr] = train(net,inputs,targets); outputs = sim(net,target)'; outputs1 = sim(net,targetp')'

Answer 1

[Greg Heath]

What previous question and answers?

What is the dimensionality of your data?

If you only have 18 samples, you probably can't afford to have 2 hidden layers. One hidden layer is always sufficient if you have enough data to estimate accurate weights. Only use more hidden layers if you have a priori information that suggests it will help.

If the dimensionality of your inputs and targets are I and O and you want to design a MLP with I-H-O topology:

[I Ntrn] = size(ptrn)

[O Ntrn] = size(ttrn)

Neq = Ntrn*O % No. of training equations

Nw = (I+1)*H+(H+1)}*O % No of unknown weights

If you are training to convergence, Neq >= Nw is required but Neq >> Nw is desired to mitigate measurement errors, noise and interference. Therefore, there is an upperbound

Hub = (Neq-O)/(I+O+1)

that is useful for choosing H.

If your inputs and outputs are 1-D, Ntrn=17, I = O =1 yields

H << floor(16/3) = 5

Therefore begin with H = 5 and run at least 10 different weight initialization trials. If successful, try decreasing H and repeat. The smaller H, the better. If you start with H = 1 and increase you may find that H = 5 isn't good enough and you will have wasted your time.

In that case you will have to increase H and use Bayesian regularization with MSEREG as an objective function.

Be sure you set the data division function to 'training'. Otherwise, the input data will be automatically divided into trn/val/tst subsets.

finally, there are 3 loops

for i = 1:N

    ptst = p(:,i);

    ttst = t(:,i);

    if i > 1 & i < N

       ptrn = p(:,[1:i-1,i+1:N]);

       ttrn = t(:,[1:i-1,i+1:N]);

    elseif ..

     ...

     else

     ...    

     end % endif

     for j = 1:5

         H = j

         for k = 1:Ntrials

             net = newff(...)

             net.trainParamgoal = 0.01*mean(var(t'));

             [net, tr] = train(...);

              MSE(i,j,k) = tr.perf(end);

etc

end

MSE = MSE % No semicolon

Hope this helps.

Greg

[Greg Heath]

Replace var(t') with var(ttrn')

Greg

[janelle]

I don't know how to appreciate your helps. Thank you so much for taking time from your busy schedule to answer me.

From "previous question and answers" I mean the question which other people asked and you answered.

Input size is 18*3 (18 sample,3 variables),Output size is 18*1

you said "be sure you set the data division function to 'training'"how can I do this? ,or in a better word, how can I divide my data manually in training and test set?

and how can I put initial weights manually?

[Greg Heath]

lookfor divide

help init

Hope this helps.

Greg

[Greg Heath]

Most of the network creation functions automatically initialize

the weights. Typically, there is no reason to override automatically assigned initial weights.

Hope this helps.

Greg

[janelle]

I so confused,when I put the default input of newfit,

net = newfit(P,T,2,{'tansig','linear'},'trainlm','learngdm','mse',{'fixunknowns','remconstantrows','mapminmax'},{'remconstantrows','mapminmax'},'dividerand')

I got this error

"??? Error using ==> feval

Undefined function or method 'remconstantrows' for input arguments of type 'char'.

Error in ==> network.subsasgn>calcProcessParams at 1075

processParams{i} = feval(processFcns{i},'pdefaults');

Error in ==> network.subsasgn>setInputProcessFcns at 960

processParams = calcProcessParams(processFcns);

Error in ==> network.subsasgn at 106

[net,err] = setInputProcessFcns(net,i,processFcns);

Error in ==> newff>new_5p1 at 145

net.inputs{1}.processFcns = ipf;

Error in ==> newff at 89

net = new_5p1(varargin{:});

Error in ==> newfit at 67

net = newff(varargin{:});

"

but when I don't input the default(net = newfit(P,T,2)), I didn't get any error and the program run well?!!!!!!!!!!!!!!!!!!

1.how can I change the DDF - Data division function, default = 'dividerand';which is the last input without input other default which I don't want to change?

2.can I put 'training' instead of 'dividerand'?

3.when we have only training does it in the main mfile set validation set itself?

[Greg Heath]

1/2. Use 'dividetrain'

3. I don't understand.

Greg

Answer 2

[Greg Heath]

1. There is an error in the NEWFIT documentation. 'linear' is not a valid option; replace with 'purelin'.

2. NEWFIT calls NEWFF. Both crash when 'remconstantrows' is an explicit input.

3. To run NEWFIT or NEWFF with explicit defaults:

a. Use 'purelin' instead of 'linear'.

b. Delete both occurrences of 'remconstantrows'. Before creating the net you can remove input and output rows that have zero variance.

4. What version of the NNTBX do you have? Both NEWFIT and NEWFF are now obsolete.

5. >> help newfit

 newfit Create a fitting network.

Obsoleted in R2010b NNET 7.0.  Last used in R2010a NNET 6.0.4.

>> help newff

 newff Create a feed-forward backpropagation network.

   Obsoleted in R2010b NNET 7.0.  Last used in R2010a NNET 6.0.4.

6. If you have a recent version of the Toolbox, use FITNET for regression and curvefitting and PATTERNNET for classification. (help ... and doc ...)

Hope this helps.

Greg

[Greg Heath]

If you want all defaults except for dividerand, you can create the net first, then override the defaults.

net = fitnet(P,T,2);

net.divideFcn = dividetrain;

Greg