Conflicting results of multcompare output c and interactive graph

Hello,

I am trying to do a multiple comparison using multcompare of my data (SampleData.mat) after having performed a n-way ANOVA with two grouping variables (gMat: materials, gCure:post-processing) to identify significant differences between the groups.

I am using R2019a.

So, my code looks like this:

%Load data
load('SampleData.mat');

%Preselection of data
idx = find(gTime=='48h');

%n-way ANOVA of selected data
[~,~,stats] = anovan(viab(idx),{gMat(idx),gCure(idx)},'model','linear');

%Multiple comparison
[c,~,~,gnames] = multcompare(stats,'Alpha',0.01,'Dimension', [1 2]);

This produces:

c =

    1.0000    2.0000   -0.0716    0.0310    0.1335    0.9386
    1.0000    3.0000   -0.2115   -0.1089   -0.0064    0.0056
    1.0000    4.0000       NaN       NaN       NaN    1.0000
    1.0000    5.0000   -0.0296    0.0729    0.1755    0.1239
    ...

and the following interactive graph:

So here are my 2 questions:

1. The interactive graph of multcompare shows that the groups 1 (X=FPMould,X2=Otoflash) and 3 (X=PlasGRAY,X2=Otoflash) are NOT significantly different. However, column 6 of c says that the p-Value is 0.0056 which is smaller than my alpha (0.01), thus suggesting that they ARE different. So, which one is correct now?
2. Is it possible that column 6 of output 'c' of multcompare does not change when I change alpha to e.g. 0.05?

Has anyone encountered this before? How can I avoid this?

Help on this would be highly appreciated!!!

Thanks,

Nina

Answer 1

[TARUN]

Hi @Nina Compera,

The mismatch you're seeing happens because the p-values in column 6 of the ‘multcompare’ output (c matrix) do not depend on the ‘Alpha’ value you set. The ‘Alpha’ parameter only affects the:

1. Width of the confidence intervals. (columns 3–5)
2. And what gets visually highlighted in the plot.

So, even if the plot doesn't clearly show a significant difference between two groups, you should trust the p-value in column 6.

In your case, a p-value of 0.0056 is below the alpha of 0.01, meaning the groups aresignificantly different, regardless of how the plot appears. To avoid confusion, rely on the p-values for significance, and use the confidence intervals for additional insight.

Please refer to the following documentation to learn more:multcompare: https://www.mathworks.com/help/stats/multcompare.html