mafdr: Interpreting Q values vs. BHFDR adjusted p-values

Using mafdr to produce false discovery rate adjusted Q values from lists of p-values has been working well for me with large datasets. The adjusted values appear reasonable. However, with very small datasets the Q values produced can be smaller than the initial p-values - particularly if many of the p-values are small. This seems wrong. As Q values are interpreted as p-values adjusted for the false discovery rate, shouldn't they always be larger than the initial p-value?

e.g.

if true
  >> P

P =

    0.0162    0.0322    0.0888    0.0495    0.0507    0.1583

>> [FDR, Q]=mafdr(P)

FDR =

    0.0023    0.0023    0.0025    0.0023    0.0018    0.0037

Q =

    0.0018    0.0018    0.0025    0.0018    0.0018    0.0037

end

A workaround for this is the 'BHFDR' option, which produces resonable looking adjustments to the p-values. It appears to use a different procedure to calculate the values

if true
 >> mafdr(P,'BHFDR', true)

ans =

    0.0761    0.0761    0.1065    0.0761    0.0761    0.1583
  end

Does anyone know why this occurs? Am I misinterpreting the meaning of the Q values? Should I switch over entirely to the 'BHFDR' procedure for both large and small datasets? Best regards, Kevin

[Kevin Casey]

To answer my own question, it looks like the Storey procedure falls apart when the list of p-values is smaller than ~1000, while the BHFDR procedure is robust with shorter lists, but more conservative.

https://stat.ethz.ch/pipermail/bioconductor/2014-January/056992.html

[Bjørn Juel]

Thank you for this. I was running into the same sorts of problems when trying to use FDR to correct for the multiple comparison.

[Samaneh Nemati]

Thank you for this. I wonder if you know what is the difference between fdr and and q that fdr matlab function gives. [fdr,q]=mafdr(pvalues). I used this function and do not know the difference between these two outputs.

[Pearl]

Thank you Kevin! This is very useful!

[Mango Wang]

@Samaneh, They are quite similar. Based on my dataset, I calculate the correlation between fdr and q, The result is 1.

Answer 1

[Mango Wang]

It seems FDR is suitable for the case when the dataset/hypothesis is very large due to the principle of the inherent method. https://www.mailman.columbia.edu/research/population-health-methods/false-discovery-rate check here for reference.

[Thomas Alderson]

This method is not always sound:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3699340/

Answer 2

[Thomas Alderson]

This method sometimes produces q values smaller than p values, which is bad

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3699340/