expr1 =

Let's look at the systems of equations you're trying to solve.

Ax = [0 1 0 0; 0 -0.1 0 0.8; 0 0 -4 9; -1 -9 2 -8];

Bx = [0 0 0 0; 0 0 -0.5 0; -2 -2 0 2; 4 0 0 0];

syms a01 a02 a03 a04 a11 a12 a13 a23 a14

expr1 = Ax*Ax*Bx(:,1) == a01*Bx(:,1) + a11*Ax*Bx(:,1)

This one clearly has no solution, since 16/5 is not equal to 0.

expr2 = Ax*Ax*Bx(:,2) == a02*Bx(:,2)+a12*Ax*Bx(:,2)

Again, no solution since -16/5 is not equal to 0.

expr3 = Ax*Ax*Ax*Bx(:,3) == a03*Bx(:,3)+a13*Ax*Bx(:,3)+a23*Ax*Ax*Bx(:,3)

This one might have a solution. We can use the third equation to get the value of a_23 and then use either the first or fourth equation to get the value of a_13. We can then use the other of the fourth or first equation to check if those values of a_23 and a_13 are consistent.

a_23 = solve(expr3(3))

a_13 = solve(subs(expr3(1), a23, a_23))

check = subs(expr3(4), [a23 a13], [a_23, a_13])

simplify(check)

So no, the equations are not consistent.

expr4 = Ax*Ax*Bx(:,2) == a04*Bx(:,4)+a14*Ax*Bx(:,4)

And again, no solution because of the second equation.

So MATLAB returning the empty 0-by-1 sym for each of those systems is correct.