How to find the best performance values for multistep ahead prediction?

With narnet in a loop I am looking for the best hidden layer size for my network in terms of future predictions. I do a multistep ahead prediction with narnet and the predicted values are good (R squared > 0.8).

My problem is that the train, validation, test and closedloop performances of my network are not correlated with the R squared value, so if I do my trials or prediction for an unknown segment, then I can't decide which hidden layer size to choose.

How can I solve this problem?

Here is the correlation matrix (the rows and columns: performance, trainPerformance, valPerformance, closedLoopPerformance, testPerformance, MSE of predicted values, R squared of predicted values)

    1.0000    0.9702    0.2953    0.1610    0.0780   -0.0401    0.0401
    0.9702    1.0000    0.2507    0.1158   -0.1657   -0.0240    0.0240
    0.2953    0.2507    1.0000   -0.0626    0.1469   -0.0918    0.0918
    0.1610    0.1158   -0.0626    1.0000    0.1826    0.2678   -0.2678
    0.0780   -0.1657    0.1469    0.1826    1.0000   -0.0622    0.0622
   -0.0401   -0.0240   -0.0918    0.2678   -0.0622    1.0000   -1.0000
    0.0401    0.0240    0.0918   -0.2678    0.0622   -1.0000    1.0000

Answer 1

[Greg Heath]

To optimize the number of hidden nodes, I routinely design multiple candidates in a loop over numH hidden node values via h = Hmin:dH:Hmax. The value of Hmax <= Hub (upper bound) is determined with respect to the difference of the number of unknown weights (NID/NFD = "N"umber of "I"nput/"F"eedback "D"elays )

 Nw = ( NID*I + NFD*O + 1)*H + (H+1)*O

and number of training equations for O-dimensional "O"utput targets

 Ntrneq = Ntrn*O ~ 0.7*N*O

However, because initial weights are random I use an inner loop over Ntrials candidates that differ w.r.t. random initial weights and, most times, random data divisions. Typically, I design 100 candidates using Ntrials = 10. However, sometimes when dH > 1, I have had to add another search using a smaller dH.

Displaying Rsq values in a Ntrials-by-numH matrix yields an excellent overview of the performance dependence on H.

For examples in the NEWSGROUP and ANSWERS, try searching with Ntrials and/or Hub

Hope this helps.

Thank you for accepting my answer

Greg

P.S. See http://www.mathworks.com/matlabcentral/newsreader/view_thread/88636#225983

[Greg Heath]

For time-series: Before the search for H, it I find it extremely useful to determine "significant" values of delays using the output autocorrelation function and input/output cross-correlation function.

Search the NEWSGROUP and ANSWERS using

 greg nncorr

Hope this helps.

Greg