Hessian in mlecov compared to hessian from fmincon

Hi, I am doing custom MLE estimation using fmincon. I want to compute asymptotic covariance matrix of MLE. To do so I could use mlecov or Hessian from fmincon. I was wondering if the hessian is estimated in mlecov in the same way as in fmincon? If not, what are possible advantages of mlecov?

Since I am running a custom problem, mlecov gives me an issue about positive definiteness. I am thinking to opt for the hessian as coming from the fmincon. However, the documentation suggest that the hessian output from fmincon is not precise https://nl.mathworks.com/help/optim/ug/hessian.html . What are other possible options? I identify three:

1. Use the maximised value of parameters from fmincon and plug it in fminunc to obtain the hessian, which seems to be correctly estimated there. Update: I still find that the Hessian is not-positive definite
2. Correct the hessian in mlecov to be positive definite (though I am not sure if it is a reliable procedure) Update: In the comment below, Matt J suggests a great function to that.
3. Drop using fmincon. Since my constraints are only non-negativity of variance, transform parameters and just use fminunc. Update: Hessian from fminunc still comes out to be non positive definite. However, mlecov does the magic. Once I plugin maxima from fmincon in the yet another custom function where parameters are transformed, such the problem is unconstrained, I find that mlecov is able to give me a positive-definite Hessian.

Did any of you happenned to have the same issue?

Thanks for your help. Regards!

[Matt J]

You could also just compute the Hessian yourself, analytically.

[zym]

Thanks, it is a good point. But my problem is too complicated for the analytical solutions. Includes Kalman filter etc...

Answer 1

[Matt J]

I was wondering if the hessian is estimated in mlecov in the same way as in fmincon?

Probably, but even if not, the Hessian inversion will be sensitive to errors from finite difference computation.

You could try nearestSPD to find the nearest positive semi-definite matrix to whatever fmincon gives you,

https://www.mathworks.com/matlabcentral/fileexchange/42885-nearestspd?s_tid=ta_fx_results

Be mindful of a a few things, though,

(1) A positive semi-definite matrix is still not positive definite.

(2) The theory of approximating MLE covariance with Hessians is only valid when no constraints are active at the solution.

[zym]

Thanks a lot for this. I will look into. I might have found a workaround.

Update on my attempt:

• I used the maxima from fmincon and plugged it in the fminunc function that applies constraint using parameter transformation
• Fminunc still gives me not a positive definite Hessian
• Then I attempt to use mlecov using custom function (note that for the argument 'data' , i put some random value, as, to my understanding, mlecov does not care about data unless it is somewhat censored)

acov = mlecov(X,5,'nloglf',custnloglf)

• Finally I get good Hessian, though, I am not sure if it is correct yet

So answer to my own question, mlecov and fminunc give different Hessian. What's the difference? no idea

[Bruno Luong]

"the Hessian inversion will be sensitive to errors from finite difference computation."

According to this page, https://uk.mathworks.com/help/optim/ug/hessian.html#bsapedt

the finite different only applied for trust region algorithm or when user sets HessianApproximation to 'finite-difference'.

The other two methods use quasi-newton, bfgs approximation of the Hessian. TMW warns that they are not accurate. I believe the bfgs and lbfgs are semi-postitive by design (the lbfgs has rank limited, about 10).

I have no idea about the relationship between mlecov (not my expertise) and the Hessian. I even don't know what is the objective function that links both? It it a log-likelyhood?

If I was you I probably do othe following:

• find the parameters that maximize the function, then
• compute the Hessian by finite-difference on the gradient, which is the in turn approximate by finite different.

The Hessian obtained this way is not symmetric, so you migh symmetrize it, then eventually perform an eigen-value decomposition, then set all the negative eigen value to 0 to make a non-negative symmetric matrix.

[zym]

Thanks for this. However, trust region algorithm needs analytical gradient, which I do not have.

To obtain asymptotic covariance matric, mlecov computes Hessian using final differences. Inside the function, they write:

% Compute pure and mixed second order central differences of the
% log-likelihood function.  H is always a square matrix, regardless of
% the shape of params.  nH is the negative hessian, because loglikefun
% returns a negative log-likelihood.

[Bruno Luong]

If that is the case, I would trust the Hessian returned by mlecov much more.

The Hessian used and returned by optimlization algorithms are designed for ... optimization, meaning the main purpose is to find a correct descend direction and not too costly. So I would not use it to make any further fine analysis.