Neural Network MEMORY problem

Hello there,

I have around 500,000 (samples) x 50 (features) matrix that I would like to train with Matlab's NN. However, with use of more then 7-8 hidden nodes (in a hidden layer), I always get memory problem. I've tried with increasing the net.trainParam.mem_reduc parameter to 1,000 and above, but that didn't help.

In order to reach some acceptable recognition rate, I would need to use more than only 7-8 hidden nodes, but memory problem doesn't allow me to :-(

So, what should I do? Should I cut my data into several smaller blocks? Would that yield similar results? Is there some alternative way to increase a processing time, but lower down the needed memory (apart from the mem_reduce parameter)? Can Matlab's NN read my data from a file (row by row)... is that possible and would that help?

I assume that I can gain the best results by training the whole data as one big matrix, so I'm trying to find a way to do that, with use of at least 30 hidden nodes in a hidden layer.

Please help :-) Thank you all! Best regards, Igor

[Thomas Hansen]

try using 'trainscg' as the training function type. This use much less memory than 'trainlm'

Answer 1

[Greg Heath]

If

[ I N ] = size(input)

[O N ] = size(output)

then

Neq = N*O % No. of training equations

and for an I-H-O net

Nw = (I+1)*H+(H+1)*O. % No. of unknown weights

Choose H ~ Hub/10 where Hub is the upper bound limit obtained from imposing

Neq >= Nw:

Hub = (Neq-O)/(I+O+1)

Then Neq ~ 10*Nw.

For each net use ~ 10 random weight initializations and choose the best w.r.t. performance of nontraining data from othe groups.

Finally, the least correlated of the best can be combined into an ensemle whose output is just the average of the member nets.

Hope this helps.

Greg

[Igor]

Thanks a lot, Greg!

I was hoping you will answer my question :-)

Cheers,

Igor

Answer 2

[Greg Heath]

Is this regression or classification?

What algorithm are you using?

What do the 50 input variables represent?

How many output variables? What do they represent?

How were the input variables chosen?

Are they linearly independent?

Which inputs have significant linear correlations with other input variables?

Which inputs have insignificant linear correlations with all of the output variables?

Are the output variables linearly independent?

Which outputs have significant linear correlations with other output variables?

It is probably unnecessary to use all of the data to train one net.

For large data sets, TRAINRP is recommended over the LM and CG algorithms.

Randomly partition the data into groups of managable size.

For each group, design a net so that

Neq >> Nw

where Neq is the number of training equations and Nw is the number of unknown weights

[Greg Heath]

The input/output questions may be useful for variable subset reduction.