MATLAB Answers

How to obtain a ROC curve through cross validation on the out of fold data in cross validation?

42 views (last 30 days)
Zeynab Mousavikhamene
Zeynab Mousavikhamene on 8 Jun 2020
Answered: Divya Gaddipati on 30 Dec 2020
I am using fitcsvm and need to obtain ROC curve for the fold that is not used in training.
Here is the code:
classificationSVM = fitcsvm(...
predictors, ...
response, ...
'KernelFunction', 'linear', ...
'PolynomialOrder', [], ...
'BoxConstraint', 1, ...
'Standardize', true, ...
'ClassNames', categorical(classnames_fitcsvm),'KFold',10);
[fitPosteriorSVM,ScoreTransform] = fitPosterior(classificationSVM);
for the prediction step:
I can not use resubPredict because that is set for training data and when I use kfoldPredict I recive error. I need to find the predicted score for the fold that is not used for training in 10 fold cross validation to be able to run perfcurve.
Any idea?

Answers (1)

Divya Gaddipati
Divya Gaddipati on 30 Dec 2020
Hi,
Currently, you have to split the data into training and validation manually to generate results on validation set.
To use 10-fold cross-validation, you can fit the model on 90% of the data, and compute results for the remaining 10% of data which was not used for fitting. You can then loop over each of the 10 subsets to plot the ROC curves for individual runs.
Here is a small example on how to do that:
k = 10;
cvFolds = crossvalind('Kfold', labels, k);
for i = 1:k
testIdx = (cvFolds == i); % get indices of test instances
trainIdx = ~testIdx; % get indices training instances
classificationSVM = fitcsvm(data_features(trainIdx,:), labels(trainIdx), ...
'KernelFunction', 'linear', ...
'PolynomialOrder', [], ...
'BoxConstraint', 1, ...
'Standardize', true);
[predictions, score] = predict(svmModel, data_features(testIdx,:));
[X,Y] = perfcurve(labels(trainIdx), predictions,'versicolor');
plot(X,Y)
xlabel('False positive rate'); ylabel('True positive rate')
title('ROC curve')
end
Hope this helps!

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!