Improve performance of Neural Network, using Bayesian Regression

Hi Guys,

I'm trying to train a neural network to fit a set of data. My problem has been solved using NN's in the past, but I don't have access to the legacy models at present, so am trying to come up with something similar myself.

I've had some success using the deep learning toolbox, particularly when using the bayesian regularisation NN models. I've tried a variety of different network architectures, each with a different number of nodes in the hidden layer (I've stuck to a single HL for now). I'm trying to optimise my model, but have spotted some unexpected behaviour in the performance plots (train vs test) that I don't know how to deal with.

As the number of nodes increases, the regression plots indicate that I'm getting better agreement between the model predictions and 'ground truth' data - as expected. The performance plots indicate that the training error reduces with each epoch. Initially, the test error decreases then increases after a while; there is no validation error presented due to the way that trainbr works. FYI the data set comprises 3800 samples, each capturing 8 features. I don't expect my network to overfit unless the number of parameters is similar to the number of samples in my dataset. For the cases I've tried, the number of parameters is much less than the number of samples. However, I'm conscious that I may be overtraining my model in an effort to get a converged result.

The Matlab documentation states that the trainbr algorithm should be run until the model converges, usually indicated by the message "Maximum MU reached."

For larger networks (100+ nodes) I need a lot of iterations (5000+) for the algorithm to 'converge'. However, the performance plot (converged result below) indicate a large difference between the training and testing data:

How should I deal with this situation? I could stop the training early (e.g. 500 epochs) or use the results of the trained algorithm, which shows a large difference between test and training error. Alternatively, is there some way to tweak the parameters of the trainbr algorithm to obtain a converged solution AND better generalisation?

Any advice would be appreciated!

B