Why does my transformation matrix only maps correctly for one point?

I'm trying to warp an image by first picking 4 points from the original image and put the points into 2 vectors X and Y. I did this with the ginput function and this is what I got:

Then I define the new positions for the points I've picked and put into another 2 vectors x and y. The new positions are defined in the same sequence as the original points. Note the case of the variables; upper case X and Y are the original, lower case x and y are the new positions.

The vectors with the new positions for the points:

After which, I try to find out the projective transformation matrix to do an imtransform in Matlab. However, I'm getting weird results even though it all seems right.

This is what I've run in Matlab:

    >> imagesc(P);
    >> [X Y] = ginput(4);
    >> A = [X(1:1,1:1) Y(1:1,1:1) 1 0 0 0 -X(1:1,1:1)*x(1:1,1:1) -Y(1:1,1:1)*x(1:1,1:1);
    0 0 0 X(1:1,1:1) Y(1:1,1:1) 1 -X(1:1,1:1)*y(1:1,1:1) -Y(1:1,1:1)*y(1:1,1:1);
    X(2:2,1:1) Y(2:2,1:1) 1 0 0 0 -X(2:2,1:1)*x(2:2,1:1) -Y(2:2,1:1)*x(2:2,1:1);
    0 0 0 X(2:2,1:1) Y(2:2,1:1) 1 -X(2:2,1:1)*y(2:2,1:1) -Y(2:2,1:1)*y(2:2,1:1);
    X(3:3,1:1) Y(3:3,1:1) 1 0 0 0 -X(3:3,1:1)*x(3:3,1:1) -Y(3:3,1:1)*x(3:3,1:1);
    0 0 0 X(3:3,1:1) Y(3:3,1:1) 1 -X(3:3,1:1)*y(3:3,1:1) -Y(3:3,1:1)*y(3:3,1:1);
    X(4:4,1:1) Y(4:4,1:1) 1 0 0 0 -X(4:4,1:1)*x(4:4,1:1) -Y(4:4,1:1)*x(4:4,1:1);
    0 0 0 X(4:4,1:1) Y(4:4,1:1) 1 -X(4:4,1:1)*y(4:4,1:1) -Y(4:4,1:1)*y(4:4,1:1)];
    >> v = [x;y];
    >> u = A \ v;
    >> U = reshape([u;1], 3, 3)';
    >> w = U*[X'; Y'; ones(1,4)];
    >> w = w ./ (ones(3,1) * w(3,:))

    w =

       -0.0000   55.7987  143.6006  174.2242
             0  273.8834   93.9077  297.0000
        1.0000    1.0000    1.0000    1.0000

    >> T = maketform('projective', U');
    >> P2 = imtransform(P, T, 'XData', [0 210], 'YData', [0 297]);
    >> imagesc(P2);

The variable P in the above code is the original image read into it. The weird thing here is, the matrix w should reflect the points that I had defined in the new positions vectors lower case x and y. However, only the first column corresponds correctly to the first point in the new positions vectors. The first column shows [0 0 1]', which reflects the first position of the new positions vectors from x_1 and y_1 [0 0]. But, the rest are off.

This causes the eventual transformation to be incorrect. Why am I not getting a correct transformation matrix?