Saving status and restart fminsearch

Hi,

so here's the situation. I use fminsearch to optimize a function. The computation of this function is long so each iteration takes a while. (The function itself is pretty simple.) The total optimization process usually takes about 1 month. The problem is that, once in a while, there is a power shortage, or some other stuff that shuts downs my computer !

I'm looking for a way to save the status of fminsearch in order to "restart it where it was". From what I understand, there is no automated way to do this (although it could be added in future releases!). I've looked at the code and, from what I understand, it might be possible to save a few matrices (x, v and fv, if I'm not mistaken) every once in a while. My problem is how to restart fminsearch with specifically those matrices...

any suggestions ?

thanks,

Vincent btw, I'm using Matlab R2013a

Answer 1

[Matt J]

You can use fminsearch's OutputFcn option to save the results of the most recent iteration to disk.

However, I find it highly suspicious that your computation takes a full month. Fminsearch is designed for solving for a small number of variables. It won't give good results if you have more than 6 unknowns or so. Therefore, if it's taking a month to solve, I imagine that either you have huge amount of data or a highly sub-optimal implementation of your objective function. If your data size is huge, I question whether you need that much data to solve for such a small number of unknowns. Seems to me like you could throw some away in the interest of speeding things up.

[John D'Errico]

Could not have said it better. If the problem is taking that long to solve, find a better optimizer. Surely your own time is worth something to you.

[Vincent]

Hi !

thanks for the suggestion about the OutputFct option ! With respect to the optimizer, the problem is not that the solution is "hard" to find for fminsearch, but that, given x, the value of f(x) is long to find (it has no explicit form and HAS to be simulated... actually using a MEX file pointing to the simulation code in Fortran...).

best,

Vincent

[Matt J]

But again, it is suspicious that the simulation would take that long, assuming that x contains a small number of unknowns and you are using a proportionately small amount of data to estimate it.

[Vincent]

I know ! It might be suspicious, but believe me, if I knew how to do it faster, I would ;)

thanks again

Vincent

[Brodie Lawson]

Everyone says that fminsearch is not recommended for problems in more than say 5 variables, but I have been working with the training of Gaussian Processes and have found that a repeatedly re-started fminsearch (it can escape small local minima when restarted, presumably due to the initial polytope being selected larger) outperforms all other traditional and stochastic methods for minimisation, even in problems with say 25 variables (11x2 effect lengths + 2 process variances + 1 data variance for fitting when using a pair of squared exponentials as the covariance function). Still exploring this issue but I would not be quick to count out fminsearch on larger scale problems when you need to avoid local minima.

Edit: Oops, didn't check the date before responding.

[Matt J]

have found that a repeatedly re-started fminsearch ... outperforms all other traditional and stochastic methods for minimisation

That is curious, but it doesn't seem very generally applicable. Even in the simplest quadratic minimization problem with 25 variables, I cannot reach TolFun=1e-6 with a million iterations of fminsearch+restart. Needless to say, other algorithms can do better...

    x=rand(1,25);

TolFval=1e-6;

for i=1:100
 [x,fval] = fminsearch(@(x) norm(x).^2,  x, ...
              optimset('MaxFunEvals',1e8,'MaxIter',1e4,'TolFun',1e-6) );
 if fval<TolFval, break;end
 i,
end

[John D'Errico]

Matt is completely correct here. fminsearch with restarts is not better. However, if you have a problem with many local minima then don't use fminsearch. Here a stochastic solver may be better, as they are explicitly designed to overcome that problem. In terms of convergence, fminsearch will be slow for large numbers of variables. If you are willing to wait a few months for a solution, then sure, use fminsearch on a 25 variable problem.

[Walter Roberson]

I have created an experimental optimizer that uses fminsearch for part of its work. I find that it is often useful in finding the right major "catch basin", but not so good at resolving the last few bits to find the true local optimum; it tends to dance around it.