The issue is not a problem with eig, except in our expectations of what eig can do. And those expecations are fueled by our experience that eig always seems to produce a set of linearly independent, as well as orthogonal, eigenvectors for our matrices. Great, when it works. But when does eig fail to work as we expect?
Again, we have expectations about eig, because most of the matrices we see as statisticians, as engineers, etc., are symmetric, and are even positive definite. Even the example problems we are given as students seem to be of that sort. But what happpens when A is NOT symmetric, or even triangular? What happens when the matrix has eigenvalues of multiplicity higher than 1?
The classic example has a matrix like the matrix A in my question:
A = [2 1 -5; 0 1 -2; 0 0 2]
A = 3×3
2 1 -5 0 1 -2 0 0 2
So is upper triangular, so clearly not the normal symmetric matrix we come to expect. Worse, it has an eigenvalue of multiplicity 2.
[V,D] = eig(A)
V = 3×3
1.0000 -0.7071 1.0000 0 0.7071 -0.0000 0 0 0.0000
D = 3×3
2 0 0 0 1 0 0 0 2
We see that 2 appears twice in the set of eigenvalues. This is our first clue that A falls in the class of defective matrices. We don't yet know that A is defective. Is it?
The second eigenvector listed is indeed an eigenvector, corresponding to eigenvalue 1.
[A*V(:,2) , V(:,2)*1]
ans = 3×2
-0.7071 -0.7071 0.7071 0.7071 0 0
As you can see, the second column of V is indeed an eigenvector. How about the first column of V?
[A*V(:,1) , V(:,1)*2]
ans = 3×2
2 2 0 0 0 0
And that clearly works too. Th problem arises when we look at that third column of V. If fact, down to floating point trash, it is the same thing as V(:,1).
norm(V(:,1) - V(:,3))
ans = 1.4186e-16
Here we see evidence of a defective matrix. While we see an eigenvalue of multiplicity 2, it lacks two independent eigenvectors for that eigenvalue.
Why then did the expression V \ D * V fail, generating a singular matrix warning? It failed, because V is a singular matrix. The columns of V do not span the R^3 space we normallty expect them to span. It is still true that this next result is zero, (or effectively zero, but for floating point trash):
norm(A*V - V*D)
ans = 4.4409e-16
But the point is, we do NOT have the relationship that V'*V is the identity matrix. In fact, the matrix V has rank 2. So we do not have the ability to write V\D*V and get something that makes sense.
ans = 2
And that is we we see a singular matrix warning when we try it.
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = 6.344132e-17.
ans = 3×3
2.0000 -0.7071 0.0000 0 1.0000 0.0000 0 0 2.0000
Defective matrices are not the normal, run of the mill matrix we expect to see. Certainly not so based on things like covariance matrices from statistics, or stiffness or stress & strain matrices from engineeering. What is broken here is not eig, since there does not exist a complete set of eigenvectors for that matrix. (I suppose it would be a nice feature if eig would throw out a warning message when that happens, but it does not. Such is life.)
In that Wikipedia link I gave above, we see the statement
"Any nontrivial Jordan block of size 2 × 2 or larger (that is, not completely diagonal) is defective. (A diagonal matrix is a special case of the Jordan normal form and is not defective.)"
So what does Jordan tell us here?
ans = 3×3
1 0 0 0 2 1 0 0 2
The Jordan form for A indeed has a 2x2 non-diagonal block in the lower right corner.
Finally, is the problem that A is upper triangular? We can change A simply enough, like this:
B = A;
B(3,3) = 3
B = 3×3
2 1 -5 0 1 -2 0 0 3
What would Jordan predict here?
ans = 3×3
3 0 0 0 1 0 0 0 2
Ah, so now the Jordan form is diagonal. It would predict that everything will work now. Well, I hope it will.
[Vb,Db] = eig(B)
Vb = 3×3
1.0000 -0.7071 -0.9733 0 0.7071 -0.1622 0 0 0.1622
Db = 3×3
2 0 0 0 1 0 0 0 3
Again, V clearly has rank 3.
ans = 3
But does the form Vb\Db*Vb recover B?
Vb \ Db * Vb
ans = 3×3
2.0000 -0.7071 1.2978 0 1.0000 0.4588 0 0 3.0000
Ugh. This still fails to work. What happened now? Is there a problem with eig? Again, the problem lies in our expectations. Well, maybe in YOUR expectations. My expectations are low here. In the help for eig, we see the statement:
[V,D] = eig(A) produces a diagonal matrix D of eigenvalues and
a full matrix V whose columns are the corresponding eigenvectors
so that A*V = V*D.
So at best, we can expect this result will be effectively zero:
norm(B*Vb - Vb*Db)
ans = 0
And that is all we can hope to see.