ttest2 doubt in the results between 2 different groups

Hello all,

I have a doubt using ttest2. One example is this one

A= [0.700000000000000, 0.850000000000000, 0.800000000000000, 0.950000000000000, 1, 0.850000000000000, 0.933333333333333, 0.700000000000000, 0.850000000000000, 0.708333333333333]

B= [0.350000000000000 0.850000000000000 0.850000000000000 0.700000000000000 1 0.600000000000000 1 0.400000000000000 1 0.791666666666667]

>> mean(A)

ans = 0.8342

>> mean(B)

ans = 0.7542

The means are very different, the same as std; however the ttest2 the results are not what we expected

>> [h,p]=ttest2(A',B')

h =0

p =0.3483

We cannot reject that both populations has the same distribution even the mean is very different and the std. How can it be?. I have tried all statistics and I get the same coclusion.

Thank you

Answer 1

[Jeff Miller]

The implication of the t-test result is that the difference (even though it looks big to you) could well have arisen by chance even if the populations had the same mean. (To me, that difference looks pretty small in comparison to the score-to-score variation within each group, and your samples are not very large.)

If you would like to get a better feel for this, run a little simulation. In each iteration, divide the 20 scores randomly into the two groups A and B, and compute the difference between the two random group means. Do that 1,000 times and look at the 1,000 differences that you get by chance (e.g., with a frequency distribution). You will see that differences as large as the one in your data (0.8342-0.7542) or even larger do often arise just by chance. The t-test is simply another way of assessing the influence of chance using some assumptions instead of simulation.