How to choose the appropriate trained NN?

Hi!

I have divided my data into trn/tst/val sets. The NN gives different classification accuracy at every training session.Should I choose my model simply based on the highest test set accuracy or should I average the test accuracy over several runs?

Thanks in advance.

Answer 1

[Greg Heath]

Doesn't make sense to average error rates in order to choose the best design.

Obtain multiple designs(e.g., ~100: ~10 for each of 10 choices for number of hidden nodes)

Rank them via the degree-of-freedom adjusted training set error and validation set error

Estimate generalization non-design error via the test set error

I tend to choose the smallest successful net by just looking at the three tabulations and plots of error vs # of hidden nodes.

Hope this helps

Thank you for formally accepting my answer

Greg

[TS Sharma]

Thank you Greg! Actually I am terribly confused between these two concepts: Model selection and parameter selection. Could you please throw some light on the two in the context of NNs? What I believe is, model selection is about selecting a suitable network architecture, whereas parameter selection is about deciding the best possible weights. I really have no clue how to select the best possible weights. Is it done using random trn/test/val sets and then selecting a model based on the highest test accuracy? Please also comment on model selection....I can see you have answered some of these questions above, but I am not aware of these techniques...Could you please direct me to a good reading material on this? Thanks.

[Greg Heath]

Different people have different definitions.

I associate model selection with topology, number of layers, choice of transfer functions, ...

I associate parameter selection with a selection of values consistent with topology and optimization of a performance function.

If you type the command (without the ending semicolon)

 net = patternnet

you will see the list of defaults chosen for both.

To begin with, run the help and doc examples. Model and parameter selection are automatically chosen by default. If performance is unsatisfactory just run the example Ntrials = 10 or more times to mitigate the random choice of initial weights and trn/val/tst data division. If that fails, increase the number of hidden nodes obtaining Ntrials = 10 or more designs for each candidate value of number of hidden nodes.

For a structured approach to this technique, search including the search words

 greg Ntrials.

Hope this helps.

Thank you for formally accepting my answer

Greg

[TS Sharma]

Thanks a lot!