Downsampling with hyperparameter optimization in Machine Learning

Hello all,

I wonder if it is possible to do that in matlab, otherwise it is an important limitation to perform machine learning in matlab. I want to do downsampling in the training but not in the validation set and use hyperparameters optimization. The problem is that it is not flexible and only allows a cross validation of the input data that is training data with the validation and they do automatically the splitting with cross-validation or hold out. I need to perform an downsampling in the training data but not to touch the validation data set. I am checking and I do not find any way to do that. Am I correct?

So I would like to do greed search considering the validation untouched and only do the downsampling in the training so the partitions they get automatically to modify the ones of the training.

Thank you

Answer 1

[Vijeta]

Matlab does provide some built-in functions for cross-validation and hyperparameter tuning for machine learning models. It can be challenging to perform downsampling only on the training data and not on the validation data.

One possible solution is to manually split your data into training and validation sets before performing downsampling on the training data. You can use the training set for hyperparameter tuning and then evaluate your final model on the untouched validation set.

You can also consider using custom cross-validation functions in Matlab that allow you to control the splitting of your data. For example, you can use the cvpartition function to create a custom partitioning scheme that only downsamples the training data while leaving the validation data untouched. Then, you can use this custom partitioning scheme in your cross-validation and hyperparameter tuning process.

cvpartition

[Esmeralda Ruiz Pujadas]

Hello for your answer.

Could you show me an example how to do a custom cvpartition?. It would be very helpful for me. I did not find any information about it.

Thank you

[Vijeta]

% Load your dataset
load('myData.mat');

% Set the desired downsampling factor for the training data
downsampleFactor = 2; % Downsample by a factor of 2

% Create a custom partitioning scheme using cvpartition
numObservations = size(X, 1);
cv = cvpartition(numObservations, 'Holdout', 0.2); % Create a holdout partition
trainIdx = cv.training; % Get the indices of the training data
trainDownsampledIdx = trainIdx(1:downsampleFactor:end); % Downsample the training indices
cvTrain = cvpartition(trainDownsampledIdx, 'Holdout', 0.2); % Create a holdout partition for the downsampled training data
partition = struct('train', trainDownsampledIdx(cvTrain.training), 'validation', find(cv.test));

% Check that the partitioning scheme has been created correctly
disp(partition);

https://www.mathworks.com/help/stats/cvpartition.html?s_tid=srchtitle_%2527holdout%20in%20cvpartition_1