How to obtain a ROC curve through cross validation on the out of fold data in cross validation?

I am using fitcsvm and need to obtain ROC curve for the fold that is not used in training.

Here is the code:

classificationSVM = fitcsvm(...
    predictors, ...
    response, ...
    'KernelFunction', 'linear', ...
    'PolynomialOrder', [], ...
    'BoxConstraint', 1, ...
    'Standardize', true, ...
    'ClassNames', categorical(classnames_fitcsvm),'KFold',10);
[fitPosteriorSVM,ScoreTransform] = fitPosterior(classificationSVM);

for the prediction step:

I can not use resubPredict because that is set for training data and when I use kfoldPredict I recive error. I need to find the predicted score for the fold that is not used for training in 10 fold cross validation to be able to run perfcurve.

Any idea?

Answer 1

[Divya Gaddipati]

Hi,

Currently, you have to split the data into training and validation manually to generate results on validation set.

To use 10-fold cross-validation, you can fit the model on 90% of the data, and compute results for the remaining 10% of data which was not used for fitting. You can then loop over each of the 10 subsets to plot the ROC curves for individual runs.

Here is a small example on how to do that:

k = 10;
cvFolds = crossvalind('Kfold', labels, k);   
for i = 1:k                                 
    testIdx = (cvFolds == i);  % get indices of test instances
    trainIdx = ~testIdx;       % get indices training instances
    
    classificationSVM = fitcsvm(data_features(trainIdx,:), labels(trainIdx), ...
        'KernelFunction', 'linear', ...
        'PolynomialOrder', [], ...
        'BoxConstraint', 1, ...
        'Standardize', true);
    [predictions, score] = predict(svmModel, data_features(testIdx,:));
    [X,Y] = perfcurve(labels(trainIdx), predictions,'versicolor');
    plot(X,Y)
    xlabel('False positive rate'); ylabel('True positive rate')
    title('ROC curve')
end

Hope this helps!