L2 Regularization Hyperparameter in trainingOptions

Hello,

I want to start training my neural network without L2 regularization.

By default, trainingOptionstrainingOptions() set the L2 regularization parameters to 1e-4, which means that it adds some penalities to the weights.

Would it be possible to train by setting L2Regularization to 0? Which is the range of values suggested for this parameter of the deepNN library?

Is there any correlation inside the library with other parameters that might be carefully tuned?

Thank you in advance for your help.

All the best.

Answer 1

[Jyothis Gireesh]

The most common values of the regularization parameter are often on a logarithmic scale between 0 and 0.1, such as 0.1, 0.001, 0.00001 etc.

Setting the regularization parameter to zero may cause the network to overfit to the training data and reduces the generalizing capability of the network. Changing the regularization parameter as such doesn’t affect other carefully tuned parameters within the model. But it’s effects may be observed during the convergence of the loss function.