You have
b=Tinv*[i ;j ;1];
x2=b(1);
y2=b(2);
if (x2<1)||(x2>M)||(y2<1)||(y2>N)
I2(i , j , Channel)=0;
else
I2(i,j ,Channel)=I1(x2 ,y2 ,Channel);
end
where Tinv is the inverse of a matrix. The inverse of a matrix is seldom exactly an integer, so your x2 and y2 are unlikely to be integers, but you use them as indices in
I2(i,j ,Channel)=I1(x2 ,y2 ,Channel);
Part of the difficulty with transforms is that any one destination typically needs to combine multiple sources because the theoretical source location is not exactly in the center of a pixel.
One of the simplest approaches is to round() the coordinates -- and to boundary-check them. This approach does produce something sort of recognizable, but it can look pretty bad, especially for steeper angles.
One of the compromise approaches is a "sub-pixel" approach. Replicate each pixel in place (for example, create a new image with each pixel replaced by a 3 x 3 block of pixels of the same value). Do the transform on the expanded matrix, possibly using round() on the coordinates, in order to create an expanded transformation of the matrix. And then somehow decide how to combine all of the resulting sub-pixels of each pixel to become a new representative pixel; that might involve taking the brightest of the subpixels, or might involve taking the median, or the mean, or might use a weighted average, or other techniques.
