how to use fminunc to get Hessian matrix

Hi,

I tried to use "fminunc" to get Hessian matrix, but received error information such as - ____________________________________________________________________________

Error using Agarch. Too many output arguments ... Error in rho1_H01_noplots (line 139) [theta,fval,exitflag,output,grad,hessian] = fminunc(@(theta)Agarch(theta,z,zx),theta0,opts);

Caused by: Failure in initial user-supplied objective function evaluation. FMINUNC cannot continue. ____________________________________________________________________________

And I defined "opts" as - ____________________________________________________________________________

opts = optimset('GradObj','on','Hessian','on','Algorithm','sqp','Display', 'off'); ____________________________________________________________________________

I have been reading the online documents relevant to "fminunc" and "hessian" and adjusted my code correspondingly, but still can't make it work. So I am wondering if anyone could help me fix this issue.

Thanks very much!

Xiao

Answer 1

[Alan Weiss]

Your options don't make sense, and may indicate a deeper problem. As explained in the options section of the fminunc documentation, the sqp algorithm does not take an input Hessian; the only algorithm that accepts an input Hessian is the trust-region algorithm. So it is possible that your objective function is trying to return the objective, gradient, and Hessian, but fminunc does not use the Hessian. So if you did not conditionalize your code to only give the number of outputs requested, then you can get this error.

That said, all algorithms can produce an output Hessian estimate.

There might be something else going on. Without more details, I don't know.

Alan Weiss

MATLAB mathematical toolbox documentation

[Sophia]

Thanks for your response. I appreciate it greatly. Here are more details about my question -

1) I want to use fminunc to estimate six coefficients and the Hessian matrix of the six estimators (without inputting any Hessian expression);

2) I prefer the estimations to be done with "quasi-newton" algorithm;

3) I made some changes based on your comments and my understanding about the online documents to my code -

_____________________________________________________________________

opts = optimset('GradObj','on','Hessian','on'); ...

[theta fval exitflag output grad hessian] = fminunc(@(theta)Agarch(theta,z,z1,z2),theta0,opts);

%"Agarch(theta,z,z1,z2)" is a function written to calculate the negative log-likelihood ____________________________________________________________________

And I received same error information as before.

Could you please point out which option(s) I didn't use correctly?

Thanks a lot for your time!

Xiao

[Alan Weiss]

You really shouldn't set your Hessian option to 'on'. If you want to use the quasi-newton algorithm, then choose it in your options.

 opts = optimset('Algorithm','quasi-newton','GradObj','on');

The main issue is your objective function. What does it return? With the GradObj option set to 'on', it should return the objective function f and the gradient of the objective g, and nothing else:

 [f,g] = Agarch(theta,z,zx)

I hope you read the section about how to conditionalize your objective function that I pointed you to above.

Alan Weiss

MATLAB mathematical toolbox documentation

[Matt J]

So it is possible that your objective function is trying to return the objective, gradient, and Hessian, but fminunc does not use the Hessian. So if you did not conditionalize your code to only give the number of outputs requested, then you can get this error.

Sounds like the opposite to me. fminunc is requesting additional outputs from the objective function like (the gradient and Hessian) required by the algorithm, but the objective function Agarch is not providing them.

[Sophia]

Dear Mr. Weiss,

I see and Thanks for your reply. Then is there any way to estimate Hessian of the estimators by using "fminunc" and "opts"? If so, could you please give me a simple example? Or, Do I have to use "fmincon" to get the output of Hessian even if I am working on unconditional estimations?

Thanks very much!

Xiao

[Alan Weiss]

I am really sorry that you have not understood the documentation. You CAN get a Hessian estimate from fminunc. Do NOT set the Hessian option, which indicates that you are supplying the Hessian!

Let's suppose that your objective function Agarch returns just the value of the objective function.

 f = Agarch(theta,z,zx) % I assume that f is a scalar

Then to get an estimated Hessian, do the following.

 opts = optimset('Algorithm','quasi-newton');
 fun = @(theta)Agarch(theta,z,zx); % I assume z and zx are in your workspace
 [theta,fval,exitflag,output,grad,hessian] = fminunc(fun,theta0,opts);

fminunc returns the Hessian matrix.

Alan Weiss

MATLAB mathematical toolbox documentation

[Sophia]

Well, Thanks very much for your help and patience, Mr. Weiss. But I did try the exactly same code as yours and received error information such as ________________________________________________________________

Error using optimset (line 230) Invalid value for OPTIONS parameter Algorithm: must be 'active-set', 'trust-region-reflective', 'interior-point', 'interior-point-convex', 'levenberg-marquardt', 'trust-region-dogleg', 'lm-line-search', or 'sqp'.

Error in R715garch11ar_1000_noplots (line 44) opts = optimset('Algorithm', 'quasi-newton'); _________________________________________________________________

I ran the code on both 2012 and 2013 versions and got same errors as above ...

Best regards,

Xiao

[Matt J]

Try instead,

    opts = optimoptions(@fminunc,'Algorithm', 'quasi-newton')

in R2013.

[Sophia]

Hi, I tried and no previous error any more, but still incorrect - __________________________________________________

Error using fminusub (line 17)

Objective function is undefined at initial point. Fminunc cannot continue.

Error in fminunc (line 419)

   [x,FVAL,GRAD,HESSIAN,EXITFLAG,OUTPUT] = fminusub(funfcn,x, ...

Error in R715garch11ar_1000_noplots (line 141)

    [theta fval exitflag output grad hessian] = fminunc(fun,theta0,opts);

___________________________________________________

Then I tried a couple of different initial values and ran the code again, but no difference yet.

I will keep on debugging my code, meanwhile, if you could help me further, that will be great. Thanks in advance.

Xiao

[Sophia]

Hi all,

Here are some updates on my question -

I was using the "true" values as the initial points for fminunc and it worked pretty well all the time; but when I tried to apply fminunc to a bootstrap procedure for my research, once a while, I got negative diagonal elements for the estimated Hessian matrix. Is it possible when applying fminunc? If so, what might be the reason(s)? Also, if I used or use some initial values very close to the "true" values for fminunc, is it reasonable for a simulation study?

Thanks very much!

Sophia

[Matt J]

Is it possible when applying fminunc? If so, what might be the reason(s)?

To study this properly, we would need you to provide code that reproduces it. As a theory, though, if the true Hessian is singular at the solution, then any numerical approximation to it will be vulnerable, due to numerical noise, of becoming slightly non-positive definite.

Also, if I used or use some initial values very close to the "true" values for fminunc, is it reasonable for a simulation study?

We're in no position to tell you what's reasonable for what you're trying to do. However, initializing with true parameter values is, I believe, a good and sensible test of best case conditions.

[Sophia]

Thanks for your response. I appreciate it.

My code is pretty long, and I am not sure if it's appropriate for me to put all of them here, but when I ran more simulations with the same code today, I did receive some error information saying that - _____________________________________________________________________

      .
      .
      .

Optimization completed because the size of the gradient is less than the default value of the function tolerance.

criteria details

Computing finite-difference Hessian using user-supplied objective function.

Undefined function or variable "relStepSize".

Error in C:\PROGRA~1\Matlab\R2013a\toolbox\optim\optim\private\lineSearch.p>bracketingPhase (line 191)

Error in C:\PROGRA~1\Matlab\R2013a\toolbox\optim\optim\private\lineSearch.p>lineSearch (line 65)

Error in fminusub (line 201) [alpha,f,grad,exitflagLnSrch,funcCountLnSrch,faultTolStruct] = ...

Error in fminunc (line 419) [x,FVAL,GRAD,HESSIAN,EXITFLAG,OUTPUT] = fminusub(funfcn,x, ...

Error in R715garch11ar_1000_noplots (line 247) [theta_star,fval_star,exitflag_star,output_star,grad_star,hessian_star] = fminunc(fun_star,theta0_star,opts);

_______________________________________________________________________

And in my code, I wrote:

_______________________________________________________

       .
       .
       .
opts = optimoptions(@fminunc,'Algorithm', 'quasi-newton');

       .
       .
       .
fun_star = @(theta_star)Agarch(theta_star,z_star,z1_star,z2_star);
%Agarch is a self-defined function used to cal the negative log-likelihood
%such as "function f = Agarch(theta,z,z1,z2) ...."
        [theta_star,fval_star,exitflag_star,output_star,grad_star,hessian_star] = fminunc(fun_star,theta0_star,opts);

       .
       .
       .
____________________________________________________

Based on the error information above, it seems to me that I need to define a variable "relStepSize" somewhere, but I don't understand what the other "Error"s mean ...

Should I define or specify more options in "opts", such as "TolX", "TolFun"?

Thanks!

[Yulong Deng]

I have the same question with you about the same error, while I tried to divide the function by some integers which can let the result show, but I also don't know how to let the original function run without any problem.