# How can I get confidence intervals when the Jacobian matrix has an almost zero vector?

11 views (last 30 days)
Tim on 14 Jan 2020
Commented: Tim on 20 Jan 2020
I am using the jacobianest function in order to evaluate the Jacobian and the confidence interval for my fitted parameters based on John D'Errico`s comment here: https://groups.google.com/forum/#!topic/comp.soft-sys.matlab/JJOqLCNJjF0
My problem that one a column of the Jacobian matrix has very small values (1e-29) and when I calculate Sigma which is proportional to inv(J'*J) (see the link above), I have a message:
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =5.516957e-117.
However, I have an excellent fit, the fitted parameters have a lot of sense but their error is so huge, it is totally unrealistic. I know that the approach above assumes that I have a Gaussian error system, but honestly, I do not know how I can be sure about it. I have a nonlinear equation with 15 parameters and I am using fminsearchcon to find the best fit with the least square method.
Is there a way to get a confidence interval using existing Matlab functions when the Gaussian assumption is not valid?
Every suggestion is welcome.

John D'Errico on 14 Jan 2020
Edited: John D'Errico on 14 Jan 2020
This is probably not a failure of the assumption of Gaussian arrors, though that may indeed be the case too. I cannot know.
What a singularity usually tells you is there is some linear combination of the parameters that does not change the sum of squares. You cannot obtain confidence intervals, since at least within some range, you can trade off any point for any other in the solution space along that line. In this case, the linear combination is a simple one.
The zero column of the Jacobian at the solution tells you that the line is one where one of the parameters is completely useless in the model. Changing it has absolutely no impact on the sum of squares. It essentially says you could fix that parameter at any value you wish. Perhaps zero is an option.
That you have an excellent fit says the model is probably adequate without needing that spurious parameter.
So remove the offending parameter from your model, as I said, fix it at your favorite value, if zero is not an option. Then redo the fit with one less parameter, and only then should you try to compute anything along the lines of a confidence interval.
Tim on 20 Jan 2020
After fixing some more parameters based on your suggestions now I have only 9 free parameters, the confidence intervals are now reasonable but the Jacobian has still its almost zero values.
I checked the rank of my Jacobian which has size(J)= [501 9], and the rank was 3, so the deficiency is quite high. However, with tolerance the rank(J,1e-20)=9 but already rank(J,1e-19)=8 only.
The smallest valued column has data in the range [-1e-25,1e-22], while the big valued column has [-1e+27,1e+29].
I am now confused a bit: what size of rank deficiency ensures that the confidence intervals are still trustworthy?
I would guess one need a full rank Jacobian....

R2016b

### Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!