The name gamma already has a meaning in MATLAB, and in fact it calculates the function you're trying to evaluate with your anonymous function gamma. For your GAMMA function see the gammainc function.
But looking at the three functions, each of the three has a problem that will cause them to error when called.
fun = @(a,t) t^(a-1) * exp(-t);
If t can be a vector, this code tries to perform matrix multiplication between those vectors and that will not work unless t was a scalar. You should use element-wise multiplication so fun can be evaluated with an array as t. Similarly, you should use element-wise power.
fun = @(a,t) t.^(a-1) .* exp(-t);
For your gamma:
gamma = @(a) integral(@(t) fun, 0, Inf);
When you call this function, integral will attempt to call your anonymous function with an input. The anonymous function will then return the anonymous function fun to integral. integral requires the function whose handle you pass in as its first input to return a double or single array, not a function handle. You need to evaluate fun inside that anonymous function. Since fun requires both a and t as inputs, you'll need to specify them both when you call fun.
gamma = @(a) integral(@(t) fun(a, t), 0, Inf);
Your GAMMA function has the same issue as your gamma function
GAMMA = @(a,x) integral(@(t) fun, x, Inf);
I also strongly encourage you not to write two functions whose names differ only in case. It would be extremely easy for you to accidentally call one when you intended to call the other. But in this case, unless you have a homework assignment that requires you to implement the gamma function or the incomplete gamma function, I also strongly encourage you just to use the gamma and/or gammainc functions included in MATLAB.
