Hello Haya,
The integral is very much defined for m=n. In that case you are integrating the function
(1/pi)*sin(n*x/2)^2*(x-pi)^2
which is well behaved and has a calculable integral. The syms expression contains a denominator
which has a factor of
and is a cubic zero when m-->n. When m--> n the numerator also must contain a factor of (n-m)^3 so that the entire expression will be a (yet to be determined) 0/0 form. To get the result you don't have to actually suss out the (n-m)^3 factor in the numerator. It's enough to know that it exists. Taking the m=n expression at the top above,
assume(n,{'positive','integer'})
f = (1/pi)*sin(n*x/2)^2*(pi-x)^2;
you obtain
pi^2/3 - (n*(pi + pi*cos(2*n*pi)) - sin(2*n*pi) + (n^2*pi^2*sin(2*n*pi))/2)/(n^3*pi)
and after setting
sin(2*n*pi) = 0 cos(2*n*pi) = 1
(there must be a way to get syms to do this but I don't know what it is) this reduces to
which doesn't work for n = 0, but that case can be ignored since the integrand is 0.