Low performance of neural network using logsig for output layer

Hi.

I have a binary classification problem and using “newff” for it. There is a single output in my neural network structure. If this outputs is equal or greater than 0.5 it belongs to class 1 and if smaller than 0.5 it belongs to class 0. So my targets for every sample is 0 or 1. Besides it I am normalizing data with “Mapstd” or “mapmaxmin”.

When I use “tansig” transfer function for hidden layer(s) and “purelin” for output, classification accuracy of network is good but when I change “purelin” to “logsig” the classification accuracy is really bad (0.5) and the classification accuracy is 50% in all repeats. What is the problem? (I'm using "Trainlm" without any normalization for outputs)

PS.

When I checked outputs after training, many of them are greater than 0.5 .

Thanks.

Answer 1

[Greg Heath]

newpr is the version of newff that is to be used for classification; newfit is for regression.

All three are obsolete (see the help and doc documentation).

Currently, patternnet is the version of feedforwardnet that is to be used for classification; fitnet is for regression.

The default output transfer function is determined according to the default output normalization function.

Therefore, for each of the two groups of 3, it is worthwhile for you to go to the help, doc and type examples in the documentation and compare the properties of the three. For the current three

 default output normalization is mapminmax [-1,1]
 default output transfer functions are tansig for patternnet and purelin for feedforwardnet and fitnet

 Therefore, changing the output function of feedforwardnet from purelin to logsig will not 
yield the correct results because the range of logsig is not {-1,1}.

 If you want to use logsig, normalized targets must be in {0,1}; The best way to do this is just to use mapminmax BEFORE train to convert targets to { 0,1 }. Then disable mapminmax during training.

 This may also be a way of using softmax. However, I'm not familiar with MATLAB's version.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Jack]

Thank you for answer Greg.

I disabled normalization and other pre-processing in network with this functions (For one hidden layer network) :

net.inputs{1}.processFcns={};
net.outputs{2}.processFcns={};

Are these codes true? (I normalized input data before inset it to neural network with Mapstd or Mapminmax).

I disabled outputs normalization because my targets are 0 and 1 so I think i dot need output normalization. (This is true?) (If this is true, why default of output normalization in "patternnet" is "mapminmax" !? All outputs are binary or vectorized binary!)

After that I set hidden layer transfer function to "Tansig" and output transfer function to "logsig", But when I set output transfer function to "purelin" the classification rate is 2~6% higher than "logsig" . Why?

PS. I don't know why but default options of patternnet doesn't have good performance in my case. I think "trainlm" is better than "trainscg" in my binary classification case. (why?)

Thanks.

[Greg Heath]

> Thank you for answer Greg.

In general, it helps when including

 1.  Version numbers of MATLAB and NNTBX
 2. Selected code to clarify explanations
 3.   Results on  a MATLAB dataset 
      help patternnet
      doc patternnet

> I disabled normalization and other pre-processing in network with this > functions (For one hidden layer network) : > > net.inputs{1}.processFcns={}; > net.outputs{2}.processFcns={}; > > Are these codes true?

Why would you ask me instead of just using them and see what happens?

 net = patternnet;                   % Using all defaults
 processFcns1 = net.inputs{1}.processFcns
 processFcns2 = net.outputs{2}.processFcns 
 net.inputs{1}.processFcns  = {}; 
 net.outputs{2}.processFcns= {}; 
 processFcns1 = net.inputs{1}.processFcns
 processFcns2 = net.outputs{2}.processFcns

> (I normalized input data before inset it to neural network with Mapstd or Mapminmax).

 Pretraining Mapstd is useful for detecting outliers. However,if you are  going to use Mapminmax why not stay with the default?

> I disabled outputs normalization because my targets are 0 and 1 so I > think i dot need output normalization. (This is true?)

 Yes, provided you use logsig or softmax.

> (If this is true, why default of output normalization in "patternnet" is "mapminmax" !? All outputs are binary or vectorized binary!)

 Probably because they didn't want to introduce an additional [0,1]   
 normalization for logsig (softmax was not available then).

I would prefer a [0,1] and logsig or softmax option for patternnet.

> After that I set hidden layer transfer function to "Tansig" and output > transfer function to "logsig", But when I set output transfer function to > "purelin" the classification rate is 2~6% higher than "logsig" . Why?

 Dunno. You didn't show your code.

 Perhaps optimal parameter settings are different.

> PS. I don't know why but default options of patternnet doesn't have good > performance in my case. I think "trainlm" is better than "trainscg" in my > binary classification case. (why?)

 Dunno. I assume they made comparisons. However, I don't know the details.

 Good Questions.

 To make my life easy I use zscore pretraining for outlier detection and 
then just accept the maxmin/tansig combination.

 I haven't used MATLAB's softmax yet ...  Having trouble getting 14a to work.