I believe that your optimization will work much better using patternsearch as the solver instead of ga, or even using surrogateopt if you have a relatively recent MATLAB version.
You can include a nonlinear constraint function to test whether the result is in the range you want. You don't need to use an output function for that.
You can also try setting your objective function to return NaN as the objective function value for infeasible points.
MATLAB mathematical toolbox documentation