It seems like you're experiencing a discrepancy between the criterion values during feature selection and the actual loss when you use the selected features to train and evaluate an SVM model. This can happen for several reasons, and I'll try to address some potential causes and solutions.
Firstly, the sequentialfs function in MATLAB performs feature selection by trying to minimize a criterion, which in your case is the classification error rate given by the loss function of the SVM. The criterion values you see during the feature selection process are based on cross-validation on your training set.
Here are some points to consider:
- The criterion values during feature selection are likely based on cross-validation and may not directly translate to the absolute performance of the model. They are relative values that help determine which features to add or remove during the selection process.
- In the first classifierfun, you're using the loss function with 'ClassifError', which computes the classification error rate (a proportion). In the second classifierfun, you're directly counting the number of misclassifications. These two approaches are different; the loss function normalizes the error by the number of test samples, while the sum of predict does not. This could explain why the criterion values are so different.
- The cross-validation process used within sequentialfs can lead to variability in the criterion values, especially if your dataset is small or if there's a high variance in your model's performance across different folds.
- Feature selection algorithms can be unstable, meaning that small changes in the data can lead to different features being selected. This is especially true for algorithms like forward selection.
- It's possible that the feature selection process is overfitting to the training data, especially if the number of features is high relative to the number of samples. This could lead to low error rates during feature selection that don't generalize well to new data.
To address these issues, you could:
- Make sure you have a separate test set that was not used during feature selection to evaluate the final model's performance.
- Use regularization techniques in your SVM to prevent overfitting.
- Use methods to increase the stability of feature selection, such as running the selection multiple times on different subsamples of the data and choosing features that are consistently selected.
- Ensure that the performance metric you use during feature selection aligns well with your model's intended use case. Sometimes, accuracy is not the best measure, and you might want to consider other metrics like F1-score, precision, recall, or AUC-ROC, depending on your problem.
- Experiment with different criteria for feature selection. For instance, you might want to use a different loss function or a different measure of model performance.
Remember that feature selection is just one part of the model-building process, and it's important to consider the overall workflow, including data preprocessing, model selection, and evaluation.
