You calculated the RMSE incorrectly -- and then had a remarkable numerical coincidence.
RMSE = sqrt(mean(((Y-Y_hat).^2)))
which is equivalent to
RMSE = sqrt(sum(((Y-Y_hat).^2)/N_obs))
where N_obs is the number of observations. (N_obs = 14 in your case.) You got the value RMSE = 0.7847.
But the correct calculation of RMSE divides by the number of degrees of freedom, not the number of observations. The correct RMSE calculation is
RMSE = sqrt(sum(((Y-Y_hat).^2)/(N_obs-rankX)))
where rankX = 3 in your case.
RMSE = sqrt(sum(((Y-Y_hat).^2)/11))
and is equal to 0.8853 (as MATLAB got).
The numerical coincidence, and complete red herring, is that this is very nearly equal to the square root of your incorrect value.
You can see where (the latest version of) MATLAB does the calculation of MSE around lines 1436-1440 of the file LinearModel.