How can I solve the problem for this code related to retraining a neural network?

Hello I have a question regarding "Neural Network Toolbox User's Guide R2013b". In this document, there are some suggested algorithms to improve generalization in which the first one is retraining neural network. In this document on page 8-36 the program is as follows: [x,t] = house_dataset; Q = size(x,2); Q1 = floor(Q*0.90); Q2 = Q-Q1; ind = randperm(Q); ind1 = ind(1:Q1); ind2 = ind(Q1+(1:Q2)); x1 = x(:,ind1); t1 = t(:,ind1); x2 = x(:,ind2); t2 = t(:,ind2); net = feedforwardnet(10); numNN = 10; NN = cell(1,numNN); perfs = zeros(1,numNN); for i=1:numNN disp(['Training ' num2str(i) '/' num2str(numNN)]) NN{i} = train(net,x1,t1); y2 = NN{i}(x2); perfs(i) = mse(net,t2,y2); end

I have problem with this part of program y2 = NN{i}(x2);. When I run this part of the program Matlab shows this error: ??? Error using ==> network.subsref at 83 Subscript indices must either be real positive integers or logical

It seems as if we want to use cell array indexing, if so, we must use indices in parenthesis instead of x2 which is 10% part of house_dataset. Am I correct in this field? if not, how can I obviate this problem to correct this code? 2. Why do not we use matlab simulation after training the network on the first part of the data set? for instance : y=sim(net,x2,t2) and then calling the performance function. 3. When we use feedforwardnet the data set are automatically divided into three parts. Should we disable this with net. divideFcn="" before running this code? Your kind help is greatly appreciated.

[Greg Heath]

Please repost using the {} (see the {} icons above the response boxes) to format your code

Answer 1

[Greg Heath]

I don't have the new documentation yet. I assume "improve generalization" means to improve the estimate of performance on nontraining data. To do so, a 10-fold crossvalidation approach is attempted with a train/test data division.

However, in doing so, important defaults that have been implemented in the software and details that have have been discussed in many postings have been ignored.

 1. minmaxxt = minmax( [ x; t ])   % Orders of magnitude differences in variable scales makes it difficult to understand the correspondence between variables and weights. Although there are normalization and un-normalization defaults internal to the training function to prevent large scale input domination and/or sigmoid saturation, it doesn't help the user's understanding, especially if the net has to be implemented in analytic form outside of MATLAB.

2. This code is just the traditional 10-fold train/test cross-validation for which there is no NNTBX function. However, to avoid overtraining an overfit net, current NNET data division is 3-fold(train/validation/test).

http://www.mathworks.com/matlabcentral/newsreader/view_thread/331993#912331 Thread Subject: NEURAL NETWORK TRAINING, VALIDATION AND TESTING

Design = Train(estimate weights) + Validation(Stop training when MSEval goes thru a minimum AND used to rank multiple designs).

Non-design = Test(Obtain UNBIASED generalization estimate of performance on unseen non-design data, sometimes using only the top-ranked of multiple nets chosen using the validation set performance).

A trn/val/tst MATLAB code is presented in

http://www.mathworks.com/matlabcentral/newsreader/view_thread/331830#911882 Thread Subject: NEURAL NET CROSSVALIDATION DESIGN EXAMPLE

3. The RNG is not initialized. Therefore results cannot be repeated.

4. ERROR: The default random trn/val/tst data division is used on the training set.

5. The output is the relatively informative unscaled test set meansquarederror = mean(variance(error')). It is more desirable to include the performance of the INDIVIDUAL trn/val/tst subsets for EACH class using normalized MSE or R-squared (see Wikipedia/R-squared)

NMSE = mean(variance(error',1)/ mean(variance(target',1))

Rsquared = 1 - NMSE.

Hope this helps.

Thank you for formally accepting my answer

Greg

Answer 2

[Greg Heath]

You may be interested in my approach. I have several hundred examples in the NEWSREADER and ANSWERS.

Just use the search word

 Ntrials

I use fitnet for regression and patternet for classification

 Ntrials   fitnet

I don't think I posted any using the house_dataset.

[Greg Heath]

Two corrections:

 NMSE = mean(variance(error',1)/ mean(variance(target',1))

 Rsquared = 1 - NMSE.

[Mohamad]

Dear Greg Heath Many thanks for the kind help.