Precision network analysis NarX

Well I think I know how to do it: in the matrix of new inputs and new targets I can go poking above the predicted outputs and by looping multiple outputs could get ahead, because as I've seen in a code, to perform 300 iterations of a one time I need to give the 300 inputs (question that is impossible because of the future inputs do not know). Now the dilemma is this: I have done an analysis of details, I've closed the loop and I checked with more data if more or less fulfilled the prediction, but unfortunately I have not been successful. For this I used about 2100 but I think data are insufficient to achieve the desired accuracy. Besides the variables are correlated but do not show much correlation (0.35 or so). In openloop get accuracy R2a = 0.82 with MSE = 0.0016 and performing the analysis of R2 in training / validation / test none of them made me overfitting. You think that you should use more data? thank you very much.

Answer 1

[Greg Heath]

1. Use original trn/val/tst data and 'divideblock' to create and evaluate the open loop net.

2. If val and tst performance is unsatisfactory, try to design better nets via changing delays, numHidden, random initial weights and/or data-split ratios. Oherwise ...

3. Close the loop and evaluate using the ORIGINAL data. Unbiased val and tst performances are much more important than the biased trn performance (even if the latter bias is mitigated by a degree-of-freedom-adjustment).

4. Predict closed loop performance using NEW test input data. If corresponding NEW target data is available, evaluate the performance.

 I think data are insufficient to achieve the desired accuracy. Besides the variables are correlated but do not show much correlation (0.35 or so).

Are you referring to zero lag correlations obtained from corrcoef or dynamic correlations obtained from nncorr, xcorr or crosscorr? Nonzero lag correlations are usually significant above 0.14 or so.

 In openloop get accuracy R2a = 0.82 with MSE = 0.0016 ...

Relatively irrelevant. What are R2val and R2tst?

 and performing the analysis of R2 in training / validation / test none of them made me overfitting.

Overfitting is usually detected by high R2trn but low R2val, R2tst and possibly R2trna.

You think that you should use more data?

If Nw << Ntrneq, you probably have enough training data.

If std(MSEtst) << MSEtst you probably have enough test data and similarly for val.

Look up the expression for std(MSE) assuming MSE is Chi-square distributed in Wikipedia (I only remember it for binomially distributed classifications).

Hope this helps.

Thank you for formally accepting my answer.

Greg

[FRANCISCO]

Thank you very much for your reply. I achieved incredible details in openloop, around 96% using highly correlated variables and doing some processing to achieve such precision. high R2 values both train, validation and test. In no case is made overfitting. I need a little more polishing the details when I close the loop, I think using a rather high value of hidden nodes significantly degrades accuracy so try to make the closeloop with a low value of hidden nodes, to fit precisely to the line checking. Aside will realize few iterations since the error increases iteration after iteration. Thank you very much.

[FRANCISCO]

One question: To assess the accuracy in closeloop do as follows:

if true
  % code

t3=t2(1,1:31); %Top 30 t2 values (targets of check)

errors = gsubtract(t3,yPredicted); %yPredicted are predicted by closeloop outputs, namely 30

t3=tonndata(t3,true,false);

MSEclosed = perform(netc,t3,yPredicted)

R2=1-MSEclosed/MSE00

end

Is this a correct way to evaluate the accuracy in closeloop?

Many thanks

[Greg Heath]

I don't know. How are your t1, t2, t3 and MSE00 defined?

What are I, N, O, Ntrn, Nval, Ntst, ID, FD, and H?

Let

 mxD = max(ID,FD) % size of delay buffer

then

1. The most unbiased approach is to

a. Only use the prior information stored in netc

b. Only use test set data that was used in no way for design

2. If Ntst is not sufficiently large to satisfy std(MSEtst) << MSEtst it may be useful to add a small bias by using the preceeding mxID vectors from the design data (val or trn if Nval = 0).

[FRANCISCO]

Well, in this case, I've reduced the sample to 250 data because before about 2200 and used the network sobreajustaba me, then I decided to try a smaller sample size. To obtain this sample:

N = 250

Neq = 250

Nw = 121

O = 1

ID = 1:2

FD = 1:2

H = 12

openloop and get results:

MSE = 6.2847 e-05

MSE00 = 0.000121

R2 = 0.974646

R2a = 0.951088

R2trn = 0.9823031

R2val = 0.89007373

R2tst = 0.92343341

The code I used is as follows: if true % code MSE = perform(net,targets,outputs);

MSEa=Neq*MSE/(Neq-Nw);

R2=1-MSE/MSE00; R2a=1-MSEa/MSE00a; MSEtrn=tr.perf(end);

MSEval=tr.vperf(end);

MSEtst=tr.tperf(end);

R2trn=1-MSEtrn/MSE00;

R2val=1-MSEval/MSE00;

R2tst=1-MSEtst/MSE00;

end

And I used

if true
  % code
net.divideFcn='divideblock';

net.divideParam.trainRatio=0.70;

net.divideParam.valRatio=0.15;

net.divideParam.testRatio=0.15;

net.trainParam.goal=0.01*MSE00;

end

I would like to have uploaded a photo of when I do because I can closeloop very close to the actual data and performing 10 iterations. Many thanks Greg