neural network hyperparameter tuning

Hello,

since there is no hyperparameter tuning function for neural network I wanted to try the bayesopt function. I tried to recreate the example here: https://de.mathworks.com/help/stats/bayesian-optimization-case-study.html. But this does not work. Is there a possibility to tune the number of hidden neurons? My code does not work...

[m,n] = size(Daten) ;
P = 0.7 ;
Training = Daten(1:round(P*m),:) ; 
Testing = Daten(round(P*m)+1:end,:);

XTrain=Training(:,1:n-1);
YTrain=Training(:,n);
XTest=Testing(:,1:n-1);
YTest=Testing(:,n);

c = cvpartition(YTrain,'KFold',10);
hiddenLayerSize=optimizableVariable('hiddenLayerSize',[0,20]);
minfn = @(z)kfoldLoss(fitnet(XTrain,YTrain,'CVPartition',c,...
    'hiddenLayerSize',z.hiddenLayerSize));

results = bayesopt(minfn,hiddenLayerSize,'IsObjectiveDeterministic',true,...
    'AcquisitionFunctionName','expected-improvement-plus');

Answer 1

[Don Mathis]

If you want a more complete workflow that also optimizes the learning rate, and tests the final model on your test set, you could try this:

% Make some data
Daten = rand(100, 3);
Daten(:,3) = Daten(:,1) + Daten(:,2) + .1*randn(100, 1);  % Minimum asymptotic error is .1
[m,n] = size(Daten) ;
% Split into train and test
P = 0.7 ;
Training = Daten(1:round(P*m),:) ; 
Testing = Daten(round(P*m)+1:end,:);
XTrain = Training(:,1:n-1);
YTrain = Training(:,n);
XTest = Testing(:,1:n-1);
YTest = Testing(:,n);
% Define a train/validation split to use inside the objective function
cv = cvpartition(numel(YTrain), 'Holdout', 1/3);
% Define hyperparameters to optimize
vars = [optimizableVariable('hiddenLayerSize', [1,20], 'Type', 'integer');
	    optimizableVariable('lr', [1e-3 1], 'Transform', 'log')];
% Optimize
minfn = @(T)kfoldLoss(XTrain', YTrain', cv, T.hiddenLayerSize, T.lr);
results = bayesopt(minfn, vars,'IsObjectiveDeterministic', false,...
    'AcquisitionFunctionName', 'expected-improvement-plus');
T = bestPoint(results)
% Train final model on full training set using the best hyperparameters
net = feedforwardnet(T.hiddenLayerSize, 'traingd');
net.trainParam.lr = T.lr;
net = train(net, XTrain', YTrain');
% Evaluate on test set and compute final rmse
ypred = net(XTest');
finalrmse = sqrt(mean((ypred - YTest').^2))

function rmse = kfoldLoss(x, y, cv, numHid, lr)
% Train net.
net = feedforwardnet(numHid, 'traingd');
net.trainParam.lr = lr;
net = train(net, x(:,cv.training), y(:,cv.training));
% Evaluate on validation set and compute rmse
ypred = net(x(:, cv.test));
rmse = sqrt(mean((ypred - y(cv.test)).^2));
end

[Dimitri]

Hello,

Now everything works. Do you happen to know why? The calculation of the rmse has nothing to do with training, right?

Best regards,

Dimitri

[Don Mathis]

Right, but rmse is the objective function being optimized by bayesopt. I think training was succeeding, but the final test rmse calculation was broken.

[Ali]

Can anyone answer this problem?

Its similar but for cell data

https://au.mathworks.com/matlabcentral/answers/508348-hyperparameter-optimization-bayesopt-for-cell-data

[Ali]

Can anyone answer this problem?

Its similar but for cell data

https://au.mathworks.com/matlabcentral/answers/508348-hyperparameter-optimization-bayesopt-for-cell-data

[Saeed Magsi]

Thank you @Don Mathisfor your solution. I tried this solution on my data but it has given me an error [" The Logical indices in position 2 contain a true value outside of the array bounds"]. I actually have two outputs. And this solution is not working on two outputs. It only works on single output. Can you help me in solving the problem. Thanks.

[SAIF MEHDI]

Most of these solvers are single objective functions. For your problem, you need a multi objective solver. I know two of them multiobjective GA and Pareto front. You would have to go through their help documents to understand the syntax.

Answer 2

[Sean de Wolski]

This is nowhere near as easy as it should be. The shallow neural net infrastructure is old and uses row-major variables. This needs to be accounted for and you'll see it below with a ton of.' transposes. Second, you'll need to wrap around fitnet because it doesn't take in all of the options as name-value pairs like with the modern fit* functions in the statistics toolbox. Third, the training is non-deterministic unless you seed the rng yourself.

I don't understand the math behind using kfold cross validation with a neural net. Hence, I'll use holdout below which will reliably train and evaluate the network on an independent test sets.

Daten = rand(100, 3);
[m,n] = size(Daten) ;
P = 0.7 ;
Training = Daten(1:round(P*m),:) ; 
Testing = Daten(round(P*m)+1:end,:);
XTrain=Training(:,1:n-1).'; % Note transposes
YTrain=Training(:,n).';
XTest=Testing(:,1:n-1).';
YTest=Testing(:,n).';
c = cvpartition(numel(YTrain),'Holdout', 0.25);
hiddenLayerSize=optimizableVariable('hiddenLayerSize',[1,20], 'Type', 'integer');
minfn = @(z)wrapFitNet(XTrain,YTrain, 'CVPartition', c, ...
    'hiddenLayerSize',z.hiddenLayerSize);
results = bayesopt(minfn,hiddenLayerSize,'IsObjectiveDeterministic',false,...
    'AcquisitionFunctionName','expected-improvement-plus');

Wrapper function

function cvrmse = wrapFitNet(x, y, varargin)
% Handle variable inputs
ip = inputParser;
ip.addParameter('hiddenLayerSize', 20);
ip.addParameter('CVPartition', cvpartition(numel(y),'Holdout', 0.10));
parse(ip, varargin{:});
cv = ip.Results.CVPartition;
hiddensz = ip.Results.hiddenLayerSize;

% Train net.  You would adjust other hyper parameters here.
net = fitnet(hiddensz);
nets = train(net, x(:, cv.training.'), y(:, cv.training.'));

% Evaluate on test set and compute rmse
ypred = nets(x(:, cv.test.'));
cvrmse = sqrt(sum(ypred-y(cv.test.').^2)/numel(y(cv.test)));

end

Finally, if the only thing you want to optimize is hidden layer size, it may be easiest to just run a loop from 1:20 and try them all. Bayesian optimization really helps when you have many different parameters (trainfcn, etc.)

[Dimitri]

Thanks,

i've just started with hyperparameter optimization and wanted to try it with the "simplest" machine learning method. my consideration was that i use the same structure for different learning methods. Therefore bayesopt, because it also works for svm, knn etc.. but you're right, a loop is probably the easiest one.

EDIT: I have a problem, see below.

[Ali]

Can anyone answer this problem?

Its similar but for cell data

https://au.mathworks.com/matlabcentral/answers/508348-hyperparameter-optimization-bayesopt-for-cell-data

[Raghu]

I need a similar help with anfis can someone help me?

https://uk.mathworks.com/matlabcentral/answers/865980-bayesian-hyperparameter-for-anfis?s_tid=srchtitle

[Shubham Baisthakur]

Is it possible to extend this method to optimize the number of fully-connected layers as well?

Answer 3

[Dimitri]

I'm sorry to bother you again, but I'm having trouble with your code. If the code runs through I get the following answer:

Additionally he doesn't plot any curves at bayesian optimization, which probably has to do with the error. I didn't change anything in your code. Can you help me again, please?

Dimitri

[Don Mathis]

I cut & pasted Sean's code into a single .m file and it runs successfully for me. What version of MATLAB are you using? Here is his code in the single .m file. I can just copy that into a .m file and hit the Run button and it works.

Daten = rand(100, 3);
[m,n] = size(Daten) ;
P = 0.7 ;
Training = Daten(1:round(P*m),:) ; 
Testing = Daten(round(P*m)+1:end,:);
XTrain=Training(:,1:n-1).'; % Note transposes
YTrain=Training(:,n).';
XTest=Testing(:,1:n-1).';
YTest=Testing(:,n).';
c = cvpartition(numel(YTrain),'Holdout', 0.25);
hiddenLayerSize=optimizableVariable('hiddenLayerSize',[1,20], 'Type', 'integer');
minfn = @(z)wrapFitNet(XTrain,YTrain, 'CVPartition', c, ...
    'hiddenLayerSize',z.hiddenLayerSize);
results = bayesopt(minfn,hiddenLayerSize,'IsObjectiveDeterministic',false,...
    'AcquisitionFunctionName','expected-improvement-plus');

function cvrmse = wrapFitNet(x, y, varargin)
% Handle variable inputs
ip = inputParser;
ip.addParameter('hiddenLayerSize', 20);
ip.addParameter('CVPartition', cvpartition(numel(y),'Holdout', 0.10));
parse(ip, varargin{:});
cv = ip.Results.CVPartition;
hiddensz = ip.Results.hiddenLayerSize;
% Train net.  You would adjust other hyper parameters here.
net = fitnet(hiddensz);
nets = train(net, x(:, cv.training.'), y(:, cv.training.'));
% Evaluate on test set and compute rmse
ypred = nets(x(:, cv.test.'));
cvrmse = sqrt(sum(ypred-y(cv.test.').^2)/numel(y(cv.test)));
end

[Dimitri]

I use Matlab 2018b. The code from you and Sean works with randomly generated data, but apparently not with my own. Unfortunately I don't get a solution with my data. I have noticed that this may have something to do with the learning rate. How do I modify your code so that I can use "traingd" as training function and "net.trainParam.lr" as hyper parameter?

[Don Mathis]

There's a mistake in the rmse formula. Try this:

cvrmse = sqrt(mean((ypred-y(cv.test)).^2));

[Dimitri]

This works. Thank you for your support!

Dimitri

[Ali]

Can anyone answer this problem?

Its similar but for cell data

https://au.mathworks.com/matlabcentral/answers/508348-hyperparameter-optimization-bayesopt-for-cell-data

[Madushan Rathnayaka]

how do we extend this to other parameters?