# TrainingOptionsLBFGS

## Description

Use a `TrainingOptionsLBFGS`

object to set training options for the
limited-memory BFGS (L-BFGS) optimizer, including line search method and gradient and step
tolerances.

The L-BFGS algorithm [1] is a quasi-Newton method that approximates the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. Use the L-BFGS algorithm for small networks and data sets that you can process in a single batch.

## Creation

Create a `TrainingOptionsLBFGS`

object by using the `trainingOptions`

function and specifying `"lbfgs"`

as the first
input argument.

## Properties

### L-BFGS

`MaxIterations`

— Maximum number of iterations

`1000`

(default) | positive integer

Maximum number of iterations to use for training, specified as a positive integer.

The L-BFGS solver is a full-batch solver, which means that it processes the entire training set in a single iteration.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`LineSearchMethod`

— Method to find suitable learning rate

`"weak-wolfe"`

(default) | `"strong-wolfe"`

| `"backtracking"`

Method to find suitable learning rate, specified as one of these values:

`"weak-wolfe"`

— Search for a learning rate that satisfies the weak Wolfe conditions. This method maintains a positive definite approximation of the inverse Hessian matrix.`"strong-wolfe"`

— Search for a learning rate that satisfies the strong Wolfe conditions. This method maintains a positive definite approximation of the inverse Hessian matrix.`"backtracking"`

— Search for a learning rate that satisfies sufficient decrease conditions. This method does not maintain a positive definite approximation of the inverse Hessian matrix.

`HistorySize`

— Number of state updates to store

`10`

(default) | positive integer

Number of state updates to store, specified as a positive integer. Values between 3 and 20 suit most tasks.

The L-BFGS algorithm uses a history of gradient calculations to approximate the Hessian matrix recursively. For more information, see Limited-Memory BFGS.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`InitialInverseHessianFactor`

— Initial value that characterizes approximate inverse Hessian matrix

`1`

(default) | positive scalar

Initial value that characterizes the approximate inverse Hessian matrix, specified as a positive scalar.

To save memory, the L-BFGS algorithm does not store and invert the dense Hessian matrix
*B*. Instead, the algorithm uses the approximation $${B}_{k-m}^{-1}\approx {\lambda}_{k}I$$, where *m* is the history size, the inverse Hessian
factor $${\lambda}_{k}$$ is a scalar, and *I* is the identity matrix. The
algorithm then stores the scalar inverse Hessian factor only. The algorithm updates the
inverse Hessian factor at each step.

The initial inverse hessian factor is the value of $${\lambda}_{0}$$.

For more information, see Limited-Memory BFGS.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`MaxNumLineSearchIterations`

— Maximum number of line search iterations

`20`

(default) | positive integer

Maximum number of line search iterations to determine the learning rate, specified as a positive integer.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`GradientTolerance`

— Relative gradient tolerance

`1e-5`

(default) | positive scalar

Relative gradient tolerance, specified as a positive scalar.

The software stops training when the relative gradient is less than or equal to `GradientTolerance`

.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`StepTolerance`

— Step size tolerance

`1e-5`

(default) | positive scalar

Step size tolerance, specified as a positive scalar.

The software stops training when the step that the algorithm takes is less than or equal to
`StepTolerance`

.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`InitialStepSize`

— Initial step size

`[]`

(default) | `"auto"`

| real finite scalar

*Since R2024b*

Initial step size, specified as one of these values:

`[]`

— Do not use an initial step size to determine the initial Hessian approximation.`"auto"`

— Determine the initial step size automatically. The software uses an initial step size of $$\Vert {s}_{0}{\Vert}_{\infty}=\frac{1}{2}\Vert {W}_{0}{\Vert}_{\infty}+0.1$$, where*W*are the initial learnable parameters of the network._{0}Positive real scalar — Use the specified value as the initial step size $$\Vert {s}_{0}{\Vert}_{\infty}$$.

If `InitialStepSize`

is `"auto"`

or a positive real
scalar, then the software approximates the initial inverse Hessian using $${\lambda}_{0}=\frac{\Vert {s}_{0}{\Vert}_{\infty}}{\Vert \nabla J({W}_{0}){\Vert}_{\infty}}$$, where *λ _{0}* is the initial inverse
Hessian factor and $$\nabla J({W}_{0})$$ denotes the gradients of the loss with respect to the initial learnable
parameters. For more information, see Limited-Memory BFGS.

### Data Formats

`InputDataFormats`

— Description of input data dimensions

`"auto"`

(default) | string array | cell array of character vectors | character vector

Description of the input data dimensions, specified as a string array, character vector, or cell array of character vectors.

If `InputDataFormats`

is `"auto"`

, then the software uses
the formats expected by the network input. Otherwise, the software uses the specified
formats for the corresponding network input.

A data format is a string of characters, where each character describes the type of the corresponding data dimension.

The characters are:

`"S"`

— Spatial`"C"`

— Channel`"B"`

— Batch`"T"`

— Time`"U"`

— Unspecified

For example, consider an array containing a batch of sequences where the first, second,
and third dimensions correspond to channels, observations, and time steps, respectively. You
can specify that this array has the format `"CBT"`

(channel, batch,
time).

You can specify multiple dimensions labeled `"S"`

or `"U"`

.
You can use the labels `"C"`

, `"B"`

, and
`"T"`

once each, at most. The software ignores singleton trailing
`"U"`

dimensions after the second dimension.

For a neural networks with multiple inputs `net`

, specify an array of
input data formats, where `InputDataFormats(i)`

corresponds to the
input `net.InputNames(i)`

.

For more information, see Deep Learning Data Formats.

**Data Types: **`char`

| `string`

| `cell`

`TargetDataFormats`

— Description of target data dimensions

`"auto"`

(default) | string array | cell array of character vectors | character vector

Description of the target data dimensions, specified as one of these values:

`"auto"`

— If the target data has the same number of dimensions as the input data, then the`trainnet`

function uses the format specified by`InputDataFormats`

. If the target data has a different number of dimensions to the input data, then the`trainnet`

function uses the format expected by the loss function.String array, character vector, or cell array of character vectors — The

`trainnet`

function uses the data formats you specify.

A data format is a string of characters, where each character describes the type of the corresponding data dimension.

The characters are:

`"S"`

— Spatial`"C"`

— Channel`"B"`

— Batch`"T"`

— Time`"U"`

— Unspecified

For example, consider an array containing a batch of sequences where the first, second,
and third dimensions correspond to channels, observations, and time steps, respectively. You
can specify that this array has the format `"CBT"`

(channel, batch,
time).

You can specify multiple dimensions labeled `"S"`

or `"U"`

.
You can use the labels `"C"`

, `"B"`

, and
`"T"`

once each, at most. The software ignores singleton trailing
`"U"`

dimensions after the second dimension.

For more information, see Deep Learning Data Formats.

**Data Types: **`char`

| `string`

| `cell`

### Monitoring

`Plots`

— Plots to display during neural network training

`"none"`

(default) | `"training-progress"`

Plots to display during neural network training, specified as one of these values:

`"none"`

— Do not display plots during training.`"training-progress"`

— Plot training progress.

The plot shows the training and validation loss, training and validation metrics
specified by the `Metrics`

property, and additional information about
the training progress.

To programmatically open and close the training progress plot after training, use the `show`

and `close`

functions with the second output of the `trainnet`

function. You can use the `show`

function to view the training progress even if the `Plots`

training option is specified as `"none"`

.

To switch the y-axis scale to logarithmic, use the axes toolbar.

For more information about the plot, see Monitor Deep Learning Training Progress.

`Metrics`

— Metrics to monitor

`[]`

(default) | character vector | string array | function handle | `deep.DifferentiableFunction`

object* (since R2024a)* | cell array | metric object

Metrics to monitor, specified as one of these values:

Built-in metric or loss function name — Specify metrics as a string scalar, character vector, or a cell array or string array of one or more of these names:

Metrics:

`"accuracy"`

— Accuracy (also known as top-1 accuracy)`"auc"`

— Area under ROC curve (AUC)`"fscore"`

— F-score (also known as F_{1}-score)`"precision"`

— Precision`"recall"`

— Recall`"rmse"`

— Root mean squared error`"mape"`

— Mean absolute percentage error (MAPE)*(since R2024b)*

Loss functions:

`"crossentropy"`

— Cross-entropy loss for classification tasks.*(since R2024b)*`"indexcrossentropy"`

— Index cross-entropy loss for classification tasks.*(since R2024b)*`"binary-crossentropy"`

— Binary cross-entropy loss for binary and multilabel classification tasks.*(since R2024b)*`"mae"`

/`"mean-absolute-error"`

/`"l1loss"`

— Mean absolute error for regression tasks.*(since R2024b)*`"mse"`

/`"mean-squared-error"`

/`"l2loss"`

— Mean squared error for regression tasks.*(since R2024b)*`"huber"`

— Huber loss for regression tasks*(since R2024b)*

Note that setting the loss function as

`"crossentropy"`

and specifying`"index-crossentropy"`

as a metric or setting the loss function as`"index-crossentropy"`

and specifying`"crossentropy"`

as a metric is not supported.Built-in metric object — If you need more flexibility, you can use built-in metric objects. The software supports these built-in metric objects:

When you create a built-in metric object, you can specify additional options such as the averaging type and whether the task is single-label or multilabel.

Custom metric function handle — If the metric you need is not a built-in metric, then you can specify custom metrics using a function handle. The function must have the syntax

`metric = metricFunction(Y,T)`

, where`Y`

corresponds to the network predictions and`T`

corresponds to the target responses. For networks with multiple outputs, the syntax must be`metric = metricFunction(Y1,…,YN,T1,…TM)`

, where`N`

is the number of outputs and`M`

is the number of targets. For more information, see Define Custom Metric Function.`deep.DifferentiableFunction`

object*(since R2024a)*— Function object with custom backward function. For more information, see Define Custom Deep Learning Operations.Custom metric object — If you need greater customization, then you can define your own custom metric object. For an example that shows how to create a custom metric, see Define Custom Metric Object. For general information about creating custom metrics, see Define Custom Deep Learning Metric Object. Specify your custom metric as the

`Metrics`

option of the`trainingOptions`

function.

If you specify a metric as a function handle, a `deep.DifferentiableFunction`

object, or a custom metric object and train the neural network using the
`trainnet`

function, then the layout of the targets that the software
passes to the metric depends on the data type of the targets, and the loss function that you
specify in the `trainnet`

function and the other metrics that you specify:

If the targets are numeric arrays, then the software passes the targets to the metric directly.

If the loss function is

`"index-crossentropy"`

and the targets are categorical arrays, then the software automatically converts the targets to numeric class indices and passes them to the metric.For other loss functions, if the targets are categorical arrays, then the software automatically converts the targets to one-hot encoded vectors and then passes them to the metric.

**Example: **`Metrics=["accuracy","fscore"]`

**Example: **`Metrics=["accuracy",@myFunction,precisionObj]`

`ObjectiveMetricName`

— Name of objective metric

`"loss"`

(default) | string scalar | character vector

*Since R2024a*

Name of objective metric to use for early stopping and returning the best network, specified as a string scalar or character vector.

The metric name must be `"loss"`

or match the name of a metric specified by
the `Metrics`

argument. Metrics specified using function handles are not
supported. To specify the `ObjectiveMetricName`

value as the name of a
custom metric, the value of the `Maximize`

property of the custom metric
object must be nonempty. For more information, see Define Custom Deep Learning Metric Object.

For more information about specifying the objective metric for early stopping, see `ValidationPatience`

. For more information about returning the best network using the objective metric, see `OutputNetwork`

.

**Data Types: **`char`

| `string`

`Verbose`

— Flag to display training progress information

`1`

(`true`

) (default) | `0`

(`false`

)

Flag to display training progress information in the command window, specified as `1`

(`true`

) or `0`

(`false`

).

When this property is `1`

(`true`

), the software displays this information:

Variable | Description |
---|---|

`Iteration` | Iteration number. |

`TimeElapsed` | Time elapsed in hours, minutes, and seconds. |

`TrainingLoss` | Training loss. |

`ValidationLoss` | Validation loss. If you do not specify validation data, then the software does not display this information. |

`GradientNorm` | Norm of the gradients. |

`StepNorm` | Norm of the steps. |

If you specify additional metrics in the training options, then
they also appear in the verbose output. For example, if you set the `Metrics`

training option to `"accuracy"`

, then the information includes the
`TrainingAccuracy`

and `ValidationAccuracy`

variables.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

| `logical`

`VerboseFrequency`

— Frequency of verbose printing

`50`

(default) | positive integer

Frequency of verbose printing, which is the number of iterations between printing to the Command Window, specified as a positive integer.

If you validate the neural network during training, then the software also prints to the command window every time validation occurs.

To enable this property, set the `Verbose`

training option to
`1`

(`true`

).

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`OutputFcn`

— Output functions

function handle | cell array of function handles

Output functions to call during training, specified as a function handle or cell array of function handles. The software calls the functions once before the start of training, after each iteration, and once when training is complete.

The functions must have the syntax `stopFlag = f(info)`

, where `info`

is a structure containing information about the training progress, and `stopFlag`

is a scalar that indicates to stop training early. If `stopFlag`

is `1`

(`true`

), then the software stops training. Otherwise, the software continues training.

The `trainnet`

function passes the output function the structure `info`

that contains these fields:

Field | Description |
---|---|

`Iteration` | Iteration number |

`TimeElapsed` | Time elapsed in hours, minutes, and seconds |

`TrainingLoss` | Training loss |

`ValidationLoss` | Validation loss. If you do not specify validation data, then the software does not display this information. |

`GradientNorm` | Norm of the gradients |

`StepNorm` | Norm of the steps |

`State` | Iteration training state, specified as `"start"` , `"iteration"` , or `"done"` . |

If you specify additional metrics in the training options, then
they also appear in the training information. For example, if you set the
`Metrics`

training option to `"accuracy"`

, then the
information includes the `TrainingAccuracy`

and
`ValidationAccuracy`

fields.

If a field is not calculated or relevant for a certain call to the output functions, then that field contains an empty array.

For an example showing how to use output functions, see Custom Stopping Criteria for Deep Learning Training.

**Data Types: **`function_handle`

| `cell`

### Validation

`ValidationData`

— Data to use for validation during training

`[]`

(default) | datastore | cell array | `minibatchqueue`

object* (since R2024a)*

Data to use for validation during training, specified as `[]`

, a datastore, a table, a cell array, or a `minibatchqueue`

object that contains the validation predictors and targets.

During training, the software uses the validation data to calculate the validation loss and
metric values. To specify the validation frequency, use the `ValidationFrequency`

training option. You can also use the validation data to
stop training automatically when the validation objective metric stops improving. By
default, the objective metric is set to the loss. To turn on automatic validation stopping,
use the `ValidationPatience`

training option.

If `ValidationData`

is `[]`

, then the software does
not validate the neural network during training.

If your neural network has layers that behave differently during prediction than during training (for example, dropout layers), then the validation loss can be lower than the training loss.

If `ValidationData`

is `[]`

, then the software does not validate the neural network during training.

Specify the validation data as a datastore, `minibatchqueue`

object, or the
cell array `{predictors,targets}`

, where `predictors`

contains the validation predictors and `targets`

contains the validation
targets. Specify the validation predictors and targets using any of the formats supported by
the `trainnet`

function.

For more information, see the input arguments of the `trainnet`

function.

`ValidationFrequency`

— Frequency of neural network validation

`50`

(default) | positive integer

Frequency of neural network validation in number of iterations, specified as a positive integer.

The `ValidationFrequency`

value is the number of iterations between
evaluations of validation metrics. To specify validation data, use the `ValidationData`

training option.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`ValidationPatience`

— Patience of validation stopping

`Inf`

(default) | positive integer

Patience of validation stopping of neural network training, specified as a positive integer or `Inf`

.

`ValidationPatience`

specifies the number of times that the objective metric on the validation set can be worse than or equal to the previous best value before neural network training stops. If `ValidationPatience`

is `Inf`

, then the values of the validation metric do not cause training to stop early. The software aims to maximize or minimize the metric, as specified by the `Maximize`

property of the metric. When the objective metric is `"loss"`

, the software aims to minimize the loss value.

The returned neural network depends on the `OutputNetwork`

training option. To return the neural network with the best validation metric value, set the `OutputNetwork`

training option to `"best-validation"`

.

*Before R2024a: The software computes the validation patience
using the validation loss value.*

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`OutputNetwork`

— Neural network to return when training completes

`"auto"`

(default) | `"last-iteration"`

| `"best-validation"`

Neural network to return when training completes, specified as one of the following:

`"auto"`

– Use`"best-validation"`

if`ValidationData`

is specified. Otherwise, use`"last-iteration"`

.`"best-validation"`

– Return the neural network corresponding to the training iteration with the best validation metric value, where the metric to optimize is specified by the`ObjectiveMetricName`

option. To use this option, you must specify the`ValidationData`

training option.`"last-iteration"`

– Return the neural network corresponding to the last training iteration.

### Regularization and Normalization

`L2Regularization`

— Factor for L_{2} regularization

`0.0001`

(default) | nonnegative scalar

Factor for L_{2} regularization (weight decay), specified as a nonnegative scalar. For more information, see L2 Regularization.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`ResetInputNormalization`

— Option to reset input layer normalization

`1`

(`true`

) (default) | `0`

(`false`

)

Option to reset input layer normalization, specified as one of the following:

`1`

(`true`

) — Reset the input layer normalization statistics and recalculate them at training time.`0`

(`false`

) — Calculate normalization statistics at training time when they are empty.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

| `logical`

`BatchNormalizationStatistics`

— Mode to evaluate statistics in batch normalization layers

`"auto"`

(default) | `"population"`

| `"moving"`

Mode to evaluate the statistics in batch normalization layers, specified as one of the following:

`"population"`

— Use the population statistics. After training, the software finalizes the statistics by passing through the training data once more and uses the resulting mean and variance.`"moving"`

— Approximate the statistics during training using a running estimate given by update steps$$\begin{array}{l}{\mu}^{*}={\lambda}_{\mu}\widehat{\mu}+(1-{\lambda}_{\mu})\mu \\ {\sigma}^{2}{}^{*}={\lambda}_{{\sigma}^{2}}\widehat{{\sigma}^{2}}\text{}\text{+}\text{}\text{(1-}{\lambda}_{{\sigma}^{2}})\text{}{\sigma}^{2}\end{array}$$

where $${\mu}^{*}$$ and $${\sigma}^{2}{}^{*}$$ denote the updated mean and variance, respectively, $${\lambda}_{\mu}$$ and $${\lambda}_{{\sigma}^{2}}$$ denote the mean and variance decay values, respectively, $$\widehat{\mu}$$ and $$\widehat{{\sigma}^{2}}$$ denote the mean and variance of the layer input, respectively, and $$\mu $$ and $${\sigma}^{2}$$ denote the latest values of the moving mean and variance values, respectively. After training, the software uses the most recent value of the moving mean and variance statistics. This option supports CPU and single GPU training only.

`"auto"`

— Use the`"moving"`

option.

### Gradient Clipping

`GradientThreshold`

— Gradient threshold

`Inf`

(default) | positive scalar

Gradient threshold, specified as `Inf`

or a positive scalar. If the gradient exceeds the value of `GradientThreshold`

, then the gradient is clipped according to the `GradientThresholdMethod`

training option.

For more information, see Gradient Clipping.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

`GradientThresholdMethod`

— Gradient threshold method

`"l2norm"`

(default) | `"global-l2norm"`

| `"absolute-value"`

Gradient threshold method used to clip gradient values that exceed the gradient threshold, specified as one of the following:

`"l2norm"`

— If the L_{2}norm of the gradient of a learnable parameter is larger than`GradientThreshold`

, then scale the gradient so that the L_{2}norm equals`GradientThreshold`

.`"global-l2norm"`

— If the global L_{2}norm,*L*, is larger than`GradientThreshold`

, then scale all gradients by a factor of`GradientThreshold/`

*L*. The global L_{2}norm considers all learnable parameters.`"absolute-value"`

— If the absolute value of an individual partial derivative in the gradient of a learnable parameter is larger than`GradientThreshold`

, then scale the partial derivative to have magnitude equal to`GradientThreshold`

and retain the sign of the partial derivative.

For more information, see Gradient Clipping.

### Sequence

`SequenceLength`

— Option to pad or truncate input sequences

`"longest"`

(default) | `"shortest"`

Option to pad or truncate the input sequences, specified as one of these options:

`"longest"`

— Pad sequences to have the same length as the longest sequence. This option does not discard any data, though padding can introduce noise to the neural network.`"shortest"`

— Truncate sequences to have the same length as the shortest sequence. This option ensures that the function does not add padding, at the cost of discarding data.

To learn more about the effects of padding and truncating the input sequences, see Sequence Padding and Truncation.

`SequencePaddingDirection`

— Direction of padding or truncation

`"right"`

(default) | `"left"`

Direction of padding or truncation, specified as one of these options:

`"right"`

— Pad or truncate sequences on the right. The sequences start at the same time step and the software truncates or adds padding to the end of each sequence.`"left"`

— Pad or truncate sequences on the left. The software truncates or adds padding to the start of each sequence so that the sequences end at the same time step.

Because recurrent layers process sequence data one time step at a time, when the recurrent
layer `OutputMode`

property is `"last"`

, any padding in
the final time steps can negatively influence the layer output. To pad or truncate sequence
data on the left, set the `SequencePaddingDirection`

argument to `"left"`

.

For sequence-to-sequence neural networks (when the `OutputMode`

property is
`"sequence"`

for each recurrent layer), any padding in the first time
steps can negatively influence the predictions for the earlier time steps. To pad or
truncate sequence data on the right, set the `SequencePaddingDirection`

option to `"right"`

.

To learn more about the effects of padding and truncating sequences, see Sequence Padding and Truncation.

`SequencePaddingValue`

— Value by which to pad input sequences

`0`

(default) | scalar

Value by which to pad the input sequences, specified as a scalar.

Do not pad sequences with `NaN`

, because doing so can
propagate errors through the neural network.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

### Hardware and Acceleration

`ExecutionEnvironment`

— Hardware resource

`"auto"`

(default) | `"gpu"`

| `"cpu"`

Hardware resource, specified as one of these values:

`"auto"`

— Use a GPU if one is available. Otherwise, use the CPU.`"gpu"`

— Use the GPU. Using a GPU requires a Parallel Computing Toolbox™ license and a supported GPU device. For information about supported devices, see GPU Computing Requirements (Parallel Computing Toolbox). If Parallel Computing Toolbox or a suitable GPU is not available, then the software returns an error.`"cpu"`

— Use the CPU.

`Acceleration`

— Performance optimization

`"auto"`

(default) | `"none"`

*Since R2024a*

Performance optimization, specified as one of these values:

`"auto"`

– Automatically apply a number of optimizations suitable for the input network and hardware resources.`"none"`

– Disable all optimizations.

### Checkpoints

`CheckpointPath`

— Path for saving checkpoint neural networks

`""`

(default) | string scalar | character vector

Path for saving the checkpoint neural networks, specified as a string scalar or character vector.

If you do not specify a path (that is, you use the default

`""`

), then the software does not save any checkpoint neural networks.If you specify a path, then the software saves checkpoint neural networks to this path and assigns a unique name to each neural network. You can then load any checkpoint neural network and resume training from that neural network.

If the folder does not exist, then you must first create it before specifying the path for saving the checkpoint neural networks. If the path you specify does not exist, then the software throws an error.

**Data Types: **`char`

| `string`

`CheckpointFrequency`

— Frequency of saving checkpoint neural networks

`30`

(default) | positive integer

Frequency of saving checkpoint neural networks in iterations, specified as a positive integer.

This option only has an effect when `CheckpointPath`

is nonempty.

**Data Types: **`single`

| `double`

| `int8`

| `int16`

| `int32`

| `int64`

| `uint8`

| `uint16`

| `uint32`

| `uint64`

## Examples

### Create Training Options for the L-BFGS Optimizer

Create a set of options for training a neural network using the L-BFGS optimizer:

Determine the learn rate using the

`"strong-wolfe"`

line search method.Stop training when the relative gradient is less than or equal to

`1e-5`

.Turn on the training progress plot.

options = trainingOptions("lbfgs", ... LineSearchMethod="strong-wolfe", ... GradientTolerance=1e-5, ... Plots="training-progress")

options = TrainingOptionsLBFGS with properties: MaxIterations: 1000 HistorySize: 10 InitialInverseHessianFactor: 1 InitialStepSize: [] LineSearchMethod: 'strong-wolfe' MaxNumLineSearchIterations: 20 GradientTolerance: 1.0000e-05 StepTolerance: 1.0000e-05 SequenceLength: 'longest' CheckpointFrequency: 30 L2Regularization: 1.0000e-04 GradientThresholdMethod: 'l2norm' GradientThreshold: Inf Verbose: 1 VerboseFrequency: 50 ValidationData: [] ValidationFrequency: 50 ValidationPatience: Inf ObjectiveMetricName: 'loss' CheckpointPath: '' ExecutionEnvironment: 'auto' OutputFcn: [] Metrics: [] Plots: 'training-progress' SequencePaddingValue: 0 SequencePaddingDirection: 'right' InputDataFormats: "auto" TargetDataFormats: "auto" ResetInputNormalization: 1 BatchNormalizationStatistics: 'auto' OutputNetwork: 'auto' Acceleration: "auto"

## Algorithms

### Limited-Memory BFGS

The L-BFGS algorithm [1] is a quasi-Newton method that approximates the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. Use the L-BFGS algorithm for small networks and data sets that you can process in a single batch.

The algorithm updates learnable parameters *W* at iteration
*k+1* using the update step given by

$${W}_{k+1}={W}_{k}-{\eta}_{k}{B}_{k}^{-1}\nabla J({W}_{k}),$$

where *W _{k}* denotes the weights at iteration

*k*, $${\eta}_{k}$$ is the learning rate at iteration

*k*,

*B*is an approximation of the Hessian matrix at iteration

_{k}*k*, and $$\nabla J({W}_{k})$$ denotes the gradients of the loss with respect to the learnable parameters at iteration

*k*.

The L-BFGS algorithm computes the matrix-vector product $${B}_{k}^{-1}\nabla J({W}_{k})$$ directly. The algorithm does not require computing the inverse of
*B _{k}*.

To save memory, the L-BFGS algorithm does not store and invert the dense Hessian matrix
*B*. Instead, the algorithm uses the approximation $${B}_{k-m}^{-1}\approx {\lambda}_{k}I$$, where *m* is the history size, the inverse Hessian
factor $${\lambda}_{k}$$ is a scalar, and *I* is the identity matrix. The
algorithm then stores the scalar inverse Hessian factor only. The algorithm updates the
inverse Hessian factor at each step.

To compute the matrix-vector product $${B}_{k}^{-1}\nabla J({W}_{k})$$ directly, the L-BFGS algorithm uses this recursive algorithm:

Set $$r={B}_{k-m}^{-1}\nabla J({W}_{k})$$, where

*m*is the history size.For $$i=m,\text{\hspace{0.17em}}\dots ,\text{\hspace{0.17em}}1$$:

Let $$\beta =\frac{1}{{s}_{k-i}^{\top}{y}_{k-i}}{y}_{k-i}^{\top}r$$, where $${s}_{k-i}$$ and $${y}_{k-i}$$ are the step and gradient differences for iteration $$k-i$$, respectively.

Set $$r=r+\text{}{s}_{k-i}\text{}\left({a}_{k-i}-\beta \right)$$, where $$a$$ is derived from $$s$$, $$y$$, and the gradients of the loss with respect to the loss function. For more information, see [1].

Return $${B}_{k}^{-1}\nabla J({W}_{k})=r$$.

### Gradient Clipping

If the gradients increase in magnitude exponentially, then the training is unstable and can diverge within a few iterations. This "gradient explosion" is indicated by a training loss that goes to `NaN`

or `Inf`

. Gradient clipping helps prevent gradient explosion by stabilizing the training at higher learning rates and in the presence of outliers [2]. Gradient clipping enables networks to be trained faster, and does not usually impact the accuracy of the learned task.

There are two types of gradient clipping.

Norm-based gradient clipping rescales the gradient based on a threshold, and does not change the direction of the gradient. The

`"l2norm"`

and`"global-l2norm"`

values of`GradientThresholdMethod`

are norm-based gradient clipping methods.Value-based gradient clipping clips any partial derivative greater than the threshold, which can result in the gradient arbitrarily changing direction. Value-based gradient clipping can have unpredictable behavior, but sufficiently small changes do not cause the network to diverge. The

`"absolute-value"`

value of`GradientThresholdMethod`

is a value-based gradient clipping method.

### L_{2} Regularization

Adding a regularization term for the weights to the loss function $$E\left(\theta \right)$$ is one way to reduce overfitting [3], [4]. The regularization term is also called *weight decay*. The loss
function with the regularization term takes the form

$${E}_{R}\left(\theta \right)=E\left(\theta \right)+\lambda \Omega \left(w\right),$$

where $$w$$ is the weight vector, $$\lambda $$ is the regularization factor (coefficient), and the regularization function $$\Omega \left(w\right)$$ is

$$\Omega \left(w\right)=\frac{1}{2}{w}^{T}w.$$

Note that the biases are not regularized [4]. You can specify the regularization factor $$\lambda $$ by using the `L2Regularization`

training option. You can also specify different
regularization factors for different layers and parameters.

The loss function that the software uses for network training includes the regularization term. However, the loss value displayed in the command window and training progress plot during training is the loss on the data only and does not include the regularization term.

## References

[1] Liu, Dong C.,
and Jorge Nocedal. "On the limited memory BFGS method for large scale optimization."
*Mathematical programming* 45, no. 1 (August 1989): 503-528. https://doi.org/10.1007/BF01589116.

[2] Pascanu, R., T. Mikolov, and Y. Bengio. "On the difficulty of training recurrent neural networks". *Proceedings of the 30th International Conference on Machine Learning*. Vol. 28(3), 2013, pp. 1310–1318.

[3] Bishop, C. M. *Pattern Recognition and Machine Learning*. Springer, New York, NY, 2006.

[4] Murphy, K. P. *Machine Learning: A Probabilistic Perspective*. The MIT Press, Cambridge, Massachusetts, 2012.

## Version History

**Introduced in R2023b**

### R2024b: Monitor and plot more metrics during training

Use new and updated metric objects during training and testing.

`MAPEMetric`

— Mean absolute percentage error (MAPE)`AccuracyMetric`

with new`NumTopKClasses`

option — Top-*k*accuracy`FScoreMetric`

with new`Beta`

option — F_{β}-score

You can also directly specify these new built-in metric and loss names:

`"mape"`

— Mean absolute percentage error (MAPE)`"crossentropy"`

— Cross-entropy loss`"index-crossentropy"`

— Index cross-entropy loss`"binary-crossentropy"`

— Binary cross-entropy loss`"mse"`

/`"mean-squared-error"`

/`"l2loss"`

— Mean squared error`"mae"`

/`"mean-absolute-error"`

/`"l1loss"`

— Mean absolute error`"huber"`

— Huber loss

### R2024b: Specify initial step size for L-BFGS solver

Specify the initial step size for the L-BFGS solver using the `InitialStepSize`

argument.

### R2024a: Specify validation data using `minibatchqueue`

object

Specify validation data as a `minibatchqueue`

object using the `ValidationData`

argument.

### R2024a: Automatic performance optimization

Accelerate training with automatic performance optimization. When you train a network
using the `trainnet`

function, automatic performance optimization is
enabled by default. You can disable performance optimization by setting the
`Acceleration`

option to `"none"`

using the
`trainingOptions`

function.

### R2024a: Specify metrics as `deep.DifferentiableFunction`

object

Specify the metrics as `deep.DifferentiableFunction`

object.

### R2024a: `OutputNetwork`

default is `"auto"`

Starting in R2024a, the `OutputNetwork`

training option default value is
`"auto"`

. If you have specified validation data, then the software
returns the network corresponding to the best validation metric value. If you have not
specified validation data, then the software returns the network corresponding to the last
training iteration. If you have validation data and want to replicate the previous default,
then set `OutputNetwork`

to `"last-iteration"`

.

This change applies when using the training options with `trainnet`

only. If you are using the training options with the `trainNetwork`

function, then there is no behavior change and by default the software returns the network
corresponding to the last training iteration.

### R2024a: `OutputNetwork`

value `"best-validation-loss"`

is not recommended

Specifying `OutputNetwork`

as `"best-validation-loss"`

is
not recommended. If you have code that set `OutputNetwork`

to
`"best-validation-loss"`

, then use `"best-validation"`

instead. The software returns the network corresponding to the best validation metric value
as specified by the `ObjectiveMetricName`

option. By default, the `ObjectiveMetricName`

value is set to
`"loss"`

. This behavior applies when using the training options with
the `trainnet`

function only.

When using the training options with the `trainNetwork`

function, if
you specify `OutputNetwork`

as `"best-validation"`

, then
software always returns the network with the best validation loss value.

## See Also

`trainingOptions`

| `trainnet`

| `dlnetwork`

| `analyzeNetwork`

| Deep Network Designer

### Topics

- Train Neural Network with Tabular Data
- Solve PDE Using Physics-Informed Neural Network
- Create Simple Deep Learning Neural Network for Classification
- Retrain Neural Network to Classify New Images
- Resume Training from Checkpoint Network
- Deep Learning with Big Data on CPUs, GPUs, in Parallel, and on the Cloud
- Define Custom Training Loops, Loss Functions, and Networks

## MATLAB Command

You clicked a link that corresponds to this MATLAB command:

Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.

Select a Web Site

Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .

You can also select a web site from the following list:

## How to Get Best Site Performance

Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.

### Americas

- América Latina (Español)
- Canada (English)
- United States (English)

### Europe

- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)

- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)