What is the maximum number of variables for practical use of fminsearch?

As part of a project i'm comparing the effectiveness of different optimisation methods to minimise the cost function (and hence train) a Neural Network. I've just started an optimisation to minimise a cost function which takes in 1275 input arguments (1275 synaptic weights). I don't mind waiting a few hours, but if this is going to take days/weeks I'd rather give up now, and just say that this method is impractical.

How long would you guess this will take?

I have a 3 year old laptop running matlab R2011(a), with a 2GHz 4GB 2 core processor, on a 64 bit windows operating system. However unfortunately Matlab can not use parpool with fminsearch, so my CPU usage is at about 60%.

Thanks!

Note: I know the Neural Network Toolbox is much better and faster at training Neural Nets, but I programmed my own Neural Net from scratch, and because of the convoluted way I've done it I can't implement back propagation. I realised this too late so i'll just have to stick with this method for now.

Answer 1

[José-Luis]

The Nealder-Mead algorithm is meant to be used for low-dimensionality problems. fminsearch uses it.

Please see:

[1] Lagarias, J.C., J. A. Reeds, M. H. Wright, and P. E. Wright, "Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions," SIAM Journal of Optimization, Vol. 9 Number 1, pp. 112-147, 1998.

for more details.

So using it for 1275 is probably a very bad idea. How long it will take is very difficult to say beforehand. It depends on the computational complexity of evaluating your cost function. It also depends on your response surface. If is is smooth, then it might converge reasonably fast, if it is complex, it might get stuck calculating forever. Anyway, it will more likely get stuck in a local optimum than actually find the global optimum. It is likely that you would end up waiting longer than the age of the universe.

Answer 2

[John D'Errico]

Expect it to take YEARS.

About 6 variables is a reasonable maximum number of variables for fminsearch. Some people might go to 8 or even 10, if they have more patience than do I. But hundreds or thousands of variables? No.

[Craig]

Thanks for your answer. I'm glad I asked this question, you've saved me a lot of time!

[Craig]

Can you please suggest an alternative optimisation method which will be faster?