How do I know the number of the hidden nodes in ANN ?

Hi All

having an ANN network , in I inputs and O outputs, how do we know the number of hidden nodes? is it related to the number of hidden layers ?

when designing an I H O topology , when you set the Hmax and Hmin and dH , you can change the number of trials by changing any of the Hmin , Hmax or even dH

what is the main rule ? and advised relation between dH and Hmin ? should they be specific numbers ?

In NO book it is discussed .

Answer 1

[Greg Heath]

[I N ] = size(input)         % size("I"nput)

[O N ] = size(target)        % size("O"utput)

Ntrn   = N - 2*round(0.15*N) % Default no. of training examples ~ 0.7*N

Ntrneq = Ntrn*O              % No. of training equations 

Nw     = (I+1)*H +(H+1)*O    % No. of unknown weights for H = number of hidden nodes

Overfitting ( Nw > Ntrneq ) allows decreased performance on nontraining (e.g., val, test and unseen) data

Since H > ( Ntrneq -O )/(I+O+1) for overfitting, one training strategy is H <= Hub or preferably, H << Hub where

     Hub = -1 + ceil( (Ntrneq - O) / (I+O+1) ) % integer H

My training strategy:

Minimize the number of hidden nodes subject to the constraint that the mean-square-error is less than 1 percent of the mean target variance

     net.trainPerform.goal = 0.01*mean(var(target',1)))

Very often this is accomplished by trial and error subject to

     0 <= Hmin <= H <= Hmax <= Hub

However, sometimes it is necessary to exceed Hub. The mitigation for this is

     Validation Stopping and/or regularization (e.g., msereg or TRAINBR}

Hope this helps.

Thank you for formally accepting my answer

Greg

[farzad]

Thank you Very Much Dear Professor

There is one left question , that I 'd like to know , I don't see and find the definition of "Hidden Nodes " in ANN

[farzad]

Still I think that when we assume the H as the number of nodes , while number of nodes should be defined as a function of number of neurons, I can't understand that how can we assume an arbitrary quantity for the number of hidden nodes

[Greg Heath]

For an I-H-O MLP:

 I number of input nodes (Fan-in-units)

 H number of hidden nodes (hidden neurons)

 O number of output nodes (output neurons)

The following 2 points are very confusing to many (most?) of the inexperienced users.

 1. The term 2 layer net implies 2 neuron layers even though there are 3 layers of nodes.

 2. Even when the output layer transfer function is just a multiplication via purelin, it is still considered a neuron layer

Since every net has input and output layers, I prefer to use the terms single hidden layer net, double hidden layer net, etc

Hope this helps.

Greg

[Greg Heath]

... I can't understand that how can we assume an arbitrary quantity for the number of hidden nodes

Where did you get that idea?

Reread what I wrote above about trying to minimize H subject to a maximum constraint on MSE.

[farzad]

Thank you dear professor, to make sure that I understand it correctly, I need to discuss it graphically and also with what I have seen in the posts, to see which point I am missing , well in this photo , I should say we have 7 Hidden Neurons ?

But for example in this link : here

you have advised in the number 2 : Reduce the number of hidden nodes, H, as much as possible

and when we defined that loop containning Hmax,dH, and Hmin , we were allowed to alter Hmax , not ?

[Greg Heath]

Yes. hidden layer 1 has 5 and hiddenlayer 2 contains 2.

Clearer if you move 6 below 7 and midway between 4 and 5.

I think you are confused about Hmax..

Hub defines the maximum value for H that guarantees you do not have more unknowns than equations. The larger the ratio Ntrneq/Nw, the more robust the design will be.

Hmax defines the largest value that you are going to try. Ideally, Hmax << Hub.

However, you may need Hmax > Hub. If so, one or both of the following is recommended.

 1. Validation Stopping 
 2. Regularization

[farzad]

Thank you Dear professor

but my question is yet unresponded ,about selection of H , so if as the picture shows , we have certain number of hidden nodes , how can we select a number for H , like 10 for that ??!!!

[farzad]

When I set the H = 1 , the Nw = 15 , but when I check it with :

 net.numWeightElements
 it was = 159

so changing H , has no direct effect on the actual number of hidden layers

[Greg Heath]

One hidden layer is ALWAYS sufficient.

My equations assume no more than 1 hidden layer.

Greg