MATLAB Answers

Issue implementing custom fully connected layer - weights converging to 0

12 views (last 30 days)
Kenny Chou
Kenny Chou on 26 Jan 2021
Answered: Anshika Chaurasia on 11 Feb 2021
I'd like to implement a custimized version of the fully connected layer. Before that, I'd like to make sure I can replicate the default FullyConnectedLayer that comes with Matlab. While my implementation did not return errors, the weights the network learned were unusually small. In my simulation, the default FullyConnectedLayer returned weights mostly between [1,0]. My implementation returned weights around 0 (e.g., 10^-40). Can someone help explain why this is happening?
classdef CustomFCLayer < nnet.layer.Layer
% Custom fully connected layer, constrain weights to be positive.
properties (Learnable)
% Layer learnable parameters
Weights
Bias
end
methods
function layer = CustomFCLayer(prev_units,num_units,name,initWeights)
% layer = CustomFCLayer(numInputs,name) creates a
% fully connected layer
layer.Name = name;
layer.Description = 'Fully connected layer';
if ~exist('initWeights','var')
std = sqrt(2/(num_units+prev_units));
layer.Weights = std*rand([num_units prev_units]);
else
assert(size(initWeights,1)==num_units)
assert(size(initWeights,2)==prev_units)
layer.Weights = initWeights;
end
layer.Bias = zeros([num_units 1]);
end
function Z = predict(layer, X)
% Forward input data through the layer at prediction time and
% output the result
Z = (layer.Weights)*X+layer.Bias;
end
end
end
  1 Comment
Larry Llewellyn
Larry Llewellyn on 3 Feb 2021
I noticed at the following link the MATLAB provied fullyConnectedLayer uses 'glorot' to initialize the weights with the Glorot initializer:
https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.fullyconnectedlayer.html#mw_bad3166a-93e7-42c0-8bd2-740cd2a842ad

Sign in to comment.

Answers (1)

Anshika Chaurasia
Anshika Chaurasia on 11 Feb 2021
Hi Kenny,
As @Larry mentioned by default the weights are initialized with Glorot initializer in fully connected layers provided by MATLAB.
Refer to following link for more information about weightsInitializer in fully connected layer:
Refer to following documentation for writing custom initializeGlorot function:
Hope it helps!

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!