neuronPCA

Principal component analysis of neuron activations

Since R2022b

Description

Add-On Required: This feature requires the Deep Learning Toolbox Model Compression Library add-on.

The compressNetworkUsingProjection function uses principal component analysis (PCA) to identify the subspace of learnable parameters that result in the highest variance in neuron activations by analyzing the network activations using a data set representative of the training data. The PCA step can be computationally intensive. If you expect to compress the same network multiple times (for example, when exploring different levels of compression), then you can perform the PCA step up front using a neuronPCA object.

Depending on the network, projection configuration, and code generation libraries used (including library-free code generation), forward passes of a projected deep neural network can be faster when you deploy the network to embedded hardware.

If you prune or quantize your network, then use compression using projection after pruning and before quantization.

Creation

Syntax

npca = neuronPCA(net,mbq)

npca = neuronPCA(net,X1,...,XN)

npca = neuronPCA(___,Name=Value)

Description

npca = neuronPCA(net,mbq) computes the principal component analysis of the neuron activations in net using the data in the mini-batch queue mbq. The software analyzes the input and output activations of the projectable layers in net.

npca = neuronPCA(net,X1,...,XN) computes the principal component analysis of the neuron activations using the data in X1,...,XN, where N is the number of network inputs.

npca = neuronPCA(___,Name=Value) sets the LayerNames property or specifies additional options using one or more name-value arguments.

Input Arguments

net — Neural network

initialized dlnetwork object

Neural network, specified as an initialized dlnetwork object.

mbq — Mini-batch queue

minibatchqueue object

Mini-batch queue that outputs data for each input of the network, specified as a minibatchqueue object.

The PCA step typically works best when using the full training set. However, any dataset that is representative of the training data distribution suffices. The input data must contain two or more observations and sequences must contain two or more time steps.

Note

Padding sequences is not recommended as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

X1,...,XN — Input data

formatted dlarray | unformatted dlarray (since R2026a) | numeric array (since R2026a) | categorical array (since R2026a) | datastore (since R2026a) | cell array of numeric arrays (since R2026a) | table (since R2026a)

Input data, specified as a formatted or unformatted dlarray object, numeric array, categorical array, datastore, cell array of numeric arrays, or table.

Since R2026a, the neuronPCA function supports the same input data types as the trainnet function. You can reuse the training data, or a subset of the training data, for compression.

For more information about dlarray formats, see the fmt input argument of dlarray.

The PCA step typically works best when using the full training set. However, any dataset that is representative of the training data distribution suffices. The input data must contain two or more observations and sequences must contain two or more time steps.

Note

Padding sequences is not recommended as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: neuronPCA(mbq,SolverMode="direct") specifies to use a direct PCA solve.

SolverMode — PCA solver mode

"auto" (default) | "direct" | "iterative"

Since R2023b

PCA solver mode, specified as one of these values:

• "auto" — Automatically select PCA solver mode using network heuristics based on layer input and output sizes.

• "direct" — For all eigenvectors and eigenvalues, perform a direct solve of the in-memory covariance matrix. Use this option when the layer inputs and outputs have small numbers of activations or when the activation covariance matrix fits in memory.

• "iterative" — Perform an iterative solve that determines the fewest eigenvectors such that the cumulative sum of the corresponding eigenvalues, in decreasing order, satisfy the MinExplainedVariance option. This option does not store the covariance matrix in memory. Use this option when layer inputs or outputs have large numbers of activations or when the activation covariance matrix does not fit in memory.

NumComponentsPerIteration — Number of eigenvectors to compute per iteration

32 (default) | positive integer

Since R2023b

Number of eigenvectors to compute per iteration, specified as a positive integer.

To specify this option, the SolverMode option must be "iterative".

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

VerbosityLevel — Verbosity level

"summary" (default) | "steps" | "iterations" | "off"

Verbosity level, specified as one of these values:

• "summary" — Display a summary of the neuron PCA algorithm.

• "steps" — Display information about the steps of the neuron PCA algorithm.

• "iterations" — Display information about the iterations of the neuron PCA algorithm.

• "off" — Do not display information.

MiniBatchSize — Size of mini-batches

"auto" (default) | positive integer

Since R2026a

Size of mini-batches to use for principal component analysis, specified as a positive integer. Larger mini-batch sizes require more memory, but can lead to faster analysis.

If you specify the input data as a mini-batch queue and MiniBatchSize is set to "auto", then the software uses the MiniBatchSize property of the mini-batch queue.

If you do not specify the input data as a mini-batch queue and MiniBatchSize is set to "auto", then the software uses mini-batch size 128.

If you specify MiniBatchSize as an integer, then the software uses the specified value as the mini-batch size, even if the input data is specified as a mini-batch queue with a different mini-batch size.

Data Types: string | char | single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

ExecutionEnvironment — Hardware resource

"auto" (default) | "gpu" | "cpu"

Hardware resource, specified as one of these values:

• "auto" — Use a GPU if one is available. Otherwise, use the CPU. If net is a quantized network with the TargetLibrary property set to "none", use the CPU even if a GPU is available.

• "gpu" — Use the GPU. Using a GPU requires a Parallel Computing Toolbox™ license and a supported GPU device. For information about supported devices, see GPU Computing Requirements (Parallel Computing Toolbox). If Parallel Computing Toolbox or a suitable GPU is not available, then the software returns an error.

• "cpu" — Use the CPU.

SequenceLength — Option to pad or truncate sequences

"longest" (default) | "shortest" | positive integer

Option to pad or truncate input sequences, specified as one of these values:

• "longest" — Pad sequences in each mini-batch to have the same length as the longest sequence. This option does not discard any data, though padding can introduce noise to the neural network.

• "shortest" — Truncate sequences in each mini-batch to have the same length as the shortest sequence. This option ensures that no padding is added, at the cost of discarding data.

To learn more about the effect of padding and truncating sequences, see Sequence Padding and Truncation.

SequencePaddingDirection — Direction of padding or truncation

"right" (default) | "left"

Since R2026a

Direction of padding or truncation, specified as one of these options:

• "right" — Pad or truncate sequences on the right. The sequences start at the same time step and the software truncates or adds padding to the end of each sequence.

• "left" — Pad or truncate sequences on the left. The software truncates or adds padding to the start of each sequence so that the sequences end at the same time step.

Recurrent layers process sequence data one time step at a time, so when the recurrent layer OutputMode property is "last", any padding in the final time steps can negatively influence the layer output. To pad or truncate sequence data on the left, set the SequencePaddingDirection name-value argument to "left".

For sequence-to-sequence neural networks (when the OutputMode property is "sequence" for each recurrent layer), any padding in the first time steps can negatively influence the predictions for the earlier time steps. To pad or truncate sequence data on the right, set the SequencePaddingDirection name-value argument to "right".

To learn more about the effects of padding and truncating sequences, see Sequence Padding and Truncation.

SequencePaddingValue — Value for padding input sequences

0 (default) | scalar

Since R2026a

Value for padding the input sequences, specified as a scalar.

Do not pad sequences with NaN, because doing so can propagate errors through the neural network.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

InputDataFormats — Description of input data dimensions

"auto" (default) | string array | cell array of character vectors | character vector

Since R2026a

Description of the input data dimensions, specified as a string array, character vector, or cell array of character vectors.

If InputDataFormats is "auto", then the software uses the formats expected by the network input. Otherwise, the software uses the specified formats for the corresponding network input.

A data format is a string of characters, where each character describes the type of the corresponding data dimension.

The characters are:

• "S" — Spatial

• "C" — Channel

• "B" — Batch

• "T" — Time

• "U" — Unspecified

For example, consider an array that represents a batch of sequences where the first, second, and third dimensions correspond to channels, observations, and time steps, respectively. You can describe the data as having the format "CBT" (channel, batch, time).

You can specify multiple dimensions labeled "S" or "U". You can use the labels "C", "B", and "T" once each, at most. The software ignores singleton trailing "U" dimensions after the second dimension.

For a neural network with multiple inputs net, specify an array of input data formats, where InputDataFormats(i) corresponds to the input net.InputNames(i).

For more information, see Deep Learning Data Formats.

Data Types: char | string | cell

CategoricalInputEncoding — Encoding of categorical inputs

"integer" (default) | "one-hot"

Since R2026a

Encoding of categorical inputs, specified as one of these values:

• "integer" — Convert categorical inputs to their integer value. In this case, the network must have one input channel for each of the categorical inputs.

• "one-hot" — Convert categorical inputs to one-hot encoded vectors. In this case, the network must have numCategories channels for each of the categorical inputs, where numCategories is the number of categories of the corresponding categorical input.

Properties

LayerNames — Names of layers to analyze

string array | cell array of character vectors | character vector

Names of layers to analyze, specified as a string array, cell array of character vectors, or a character vector containing a single layer name.

The software, by default, analyzes all the layers in the network that support projection.

The compressNetworkUsingProjection and neuronPCA functions support these layers:

• convolution1dLayer (since R2024b)

• convolution2dLayer

• fullyConnectedLayer

• lstmLayer

• gruLayer

Note

Layers that share learnable parameters with other layers through weight tying do not support compression using projection.

Since R2026a, compressNetworkUsingProjection and neuronPCA support compressing layers contained inside a networkLayer object.

If you specify LayerNames to be the name of a network layer, then the software analyzes every supported layer inside the network layer.

To analyze a specific nested layer inside a network layer, specify the name of the network layer and the name of the nested layer separated by a forward slash "/". For example, the path to a layer named "nestedLayerName" in a network layer named "networkLayerName" is "networkLayerName/nestedLayerName". If there are multiple levels of nested layers, then specify the path using the form "networkLayerName1/.../networkLayerNameN/nestedLayerName".

This property is read-only after object creation. To set this property, use the corresponding name-value argument when you create the neuronPCA object.

Data Types: string | cell

ExplainedVarianceRange — Range of explained variance

Read-only: two-element row vector

Since R2023b

This property is read-only.

Range of explained variance that the principal components computed for the analyzed layers can attain, returned as a two-element row vector.

Data Types: double

LearnablesReductionRange — Range of reduction in learnables

Read-only: two-element row vector

Since R2023b

This property is read-only.

Range of reduction in learnables that the principal components computed for the analyzed layers can attain, returned as a two-element row vector.

If the minimum reduction in learnables is greater than zero, then the explained variance is the same for all learnables reduction levels below the minimum.

Data Types: double

InputEigenvalues — Eigenvalues of activations input to projectable layers

Read-only: cell array

This property is read-only.

Eigenvalues of activations input to projectable layers, specified as a cell array. Each element contains the input eigenvalues for the corresponding projectable layer in LayerNames.

Data Types: cell

InputEigenvectors — Eigenvectors of activations input to projectable layers

Read-only: cell array

This property is read-only.

Eigenvectors of activations input to projectable layers, specified as a cell array. Each element contains the input eigenvectors for the corresponding projectable layer in LayerNames.

Data Types: cell

OutputEigenvalues — Eigenvalues of activations output by projectable layers

Read-only: cell array

This property is read-only.

Eigenvalues of activations output by projectable layers, specified as a cell array. Each element contains the output eigenvalues for the corresponding projectable layer in LayerNames.

Data Types: cell

OutputEigenvectors — Eigenvectors of activations output by projectable layers

Read-only: cell array

This property is read-only.

Eigenvectors of activations output by projectable layers, specified as a cell array. Each element contains the output eigenvectors for the corresponding projectable layer in LayerNames.

Data Types: cell

Examples

Analyze Neuron Activations for Compression Using Projection

Load the pretrained network in dlnetJapaneseVowels and the training data in JapaneseVowelsTrainData.

load dlnetJapaneseVowels
load JapaneseVowelsTrainData

Create a mini-batch queue containing the training data. To create a mini-batch queue from in-memory data, convert the sequences to an array datastore.

adsXTrain = arrayDatastore(XTrain,OutputType="same");

Create the minibatchqueue object.

• Specify a mini-batch size of 16.

• Preprocess the mini-batches using the preprocessMiniBatchPredictors function, listed in the Mini-Batch Predictors Preprocessing Function section of the example.

• Specify that the output data has format "CTB" (channel, time, batch).

miniBatchSize = 16;

mbqTrain = minibatchqueue(adsXTrain, ...
    MiniBatchSize=miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatchPredictors, ...
    MiniBatchFormat="CTB");

Create the neuronPCA object.

npca = neuronPCA(net,mbqTrain);

Using solver mode "direct".
neuronPCA analyzed 2 layers: "lstm","fc"

View the properties of the neuronPCA object.

npca

npca = 
  neuronPCA with properties:

                  LayerNames: ["lstm"    "fc"]
      ExplainedVarianceRange: [0 1]
    LearnablesReductionRange: [0 0.9690]
            InputEigenvalues: {[12×1 double]  [100×1 double]}
           InputEigenvectors: {[12×12 double]  [100×100 double]}
           OutputEigenvalues: {[100×1 double]  [9×1 double]}
          OutputEigenvectors: {[100×100 double]  [9×9 double]}

Mini-Batch Predictors Preprocessing Function

The preprocessMiniBatchPredictors function preprocesses a mini-batch of predictors by extracting the sequence data from the input cell array and truncating them along the second dimension so that they have the same length.

Note: Do not pad sequence data when doing the PCA step for projection as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

function X = preprocessMiniBatchPredictors(dataX)

X = padsequences(dataX,2,Length="shortest");

end

References

[1] "Compressing Neural Networks Using Network Projection." Accessed July 20, 2023. https://www.mathworks.com/company/technical-articles/compressing-neural-networks-using-network-projection.html.

Extended Capabilities

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

The neuronPCA function supports GPU array input with these usage notes and limitations:

• This function runs on the GPU if any of these conditions are met:

  • Any of the values of the network learnable parameters inside net.Learnables.Value are dlarray objects with underlying data of type gpuArray

  • The input argument mbq outputs dlarray objects with underlying data of type gpuArray

  • The input arguments X1,...,XN are dlarray objects with underlying data of type gpuArray

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2022b

R2026a: neuronPCA supports analyzing network layers

The neuronPCA function now supports analyzing layers contained inside of networkLayer objects.

R2026a: Projection API supports additional input data types

The neuronPCA and compressNetworkUsingProjection functions now support the same input data types as the trainnet function.

Before R2026a, the input data could be a minibatchqueue object or one or multiple formatted dlarray objects.

Since R2026a, the input data can also be specified as one of these options.

• One or multiple unformatted dlarray objects

• One or multiple numeric arrays

• One or multiple categorical arrays

• Datastore

• Cell array of numeric arrays

• Table

Additionally, both neuronPCA and compressNetworkUsingProjection now support these name-value arguments:

• MiniBatchSize — Mini-batch size.

• ExecutionEnvironment — Hardware resource for compressing the neural network.

• SequenceLength — Option to pad or truncate input sequences.

• SequencePaddingDirection — Direction of padding or truncation.

• SequencePaddingValue — Value for padding the input sequences.

• InputDataFormats — Description of the input data dimensions.

• CategoricalInputEncoding — Encoding of categorical inputs.

R2024b: neuronPCA supports analyzing 1-D convolution layers

The neuronPCA function now supports analyzing convolution1dLayer objects.

R2023b: Specify PCA solver mode

Specify the PCA solver mode using the SolverMode option. To specify the number of components per iteration of an iterative PCA solve, use the NumComponentsPerIteration option.

Starting in R2023b, the software, by default, automatically selects the PCA solver mode using neural network heuristics based on layer input and output sizes. In previous versions, the software performs a direct PCA solve for all neural networks. This can lead to differences in the numbers of eigenvalues and eigenvectors in the InputEigenvalues, InputEigenvectors, OutputEigenvalues, and OutputEigenvectors properties. To reproduce the behavior of previous versions, set the SolverMode option to "direct".

R2023b: Inspect range of explained variance and reduction of learnables

Inspect the range of explained variance and reduction of learnables that the principal components of the analyzed layers can attain using the ExplainedVarianceRange and LearnablesReductionRange properties, respectively.