Significance test of ROC value with Perfcurve (obtaining the p value)         

I would like to compare two populations of data points using ROC analysis (calculating area under the curve). The ROC values are between 0.5 and 1:00. I would like to examine whether the ROC value is significantly different from 0.5. One approach is to randomly assign the data points to two populations (say 1000 times) and then check the significance. My question is that whether 'perfcurve' can perform such a significance test and provide the p value. If yes, could you please explain in detail (or with example) how the routine should be written. Thanks

Answer 1

[Ilya]

I can't explain in detail how the routine should be written because I don't understand how you are comparing the two populations. But I can give you an idea for what you could do. If you have a sufficiently recent version of perfcurve, you can use the 'NBoot' parameter to estimate the curve uncertainty by bootstrap. The 4th output from perfcurve is then a confidence interval for AUC at the test size specified by the 'Alpha' parameter. If this interval does not include 0.5, you can conclude that AUC is statistically different from 0.5 at this value of alpha.

[Marta]

Hi! Is there any equally simple way to see whether AUCs from different classifiers are significantly different? Eq (3) of http://pubs.rsna.org/doi/pdf/10.1148/radiology.148.3.6878708 suggests a simple method for this that relies on estimates of the standard deviation of the AUC. Would you estimate the std(AUC) using the confidence intervals from the bootstrapping as detailed in your answer?

Many thanks, Marta

[Ilya]

You can estimate the standard deviation of AUC using confidence intervals from bootstrap. The paper you are referring to prescribes estimating correlation between AUCs produced by two classifiers. This can be done by bootstrap too, but clearly you would need to write your own code for that.