Neural Network Output :Scaling the output range.

Hi,

The output layer of my neural network (3 layered) is using sigmoid as activation which outputs only in range [0-1]. However, if I want to train it for outputs that are beyond [0-1], say in thousands, what should I do?

For example if I want to train

input ----> output

0 0 ------> 0

0 1 ------> 1000

1000 1 ----> 1

1 1 -------> 0

My program works for AND, OR, XOR etc. As input output are all in binary.

There were some suggestion to use,

Activation:

-----------

y = lambda*(abs(x)*1/(1+exp(-1*(x))))

Derivative of activation:

-------------------------

lambda*(abs(y)*y*(1-y))

This did not converge for the mentioned training pattern. Are there any suggestion please?

Answer 1

[Greg Heath]

Hello Greg,

Thanks again for answering the question. For my case, no rigid bound,

1. INCORRECT. ALL 3 VARIABLES ARE BOUNDED:

 0 <= X1, Y <= 1000

 0<= X2 <= 1.

2. HOWEVER, SINCE THE INPUT SCALES ARE DIFFERENT BY A FACTOR OF THOUSAND, X1 AND Y SHOULD BE TRANSFORMED BY VIA LOGS AND/OR POWERS. E.G.,

X1n = LOG10( 1 + X1 ) / LOG10( 1001 ) ==> 0 <= X1n <= 1

SIMILARLY FOR Y

HOPE THIS HELPS.

GREG

[Ashikur]

Thanks Greg, you are saving people's life :)

Answer 2

[Greg Heath]

If the target has rigid bounds, scale the data to either [0,1] or [-1,1] and use either LOGSIG or TANSIG, respectively.

Otherwise, standardize to zero-mean/unit variance and use PURELIN.

To recover the original data scale, just use the reverse tranformations.

Hope this helps.

Greg

[Ashikur]

Hello Greg,

Thanks again for answering the question. For my case, no rigid bound, So I will use purelin or f(x) = x

1. Standardize what? the input? each input vector individually? or the whole training input matrix?

2. Do I need to do the same standardization for desired output too?

I used purlin (f(x)=x) in my output, but it seems not to converge for,

0 0 ------> 0

0 1 ------> 1000

1000 1 ----> 1

1 1 -------> 0

The codes are in below link,

https://docs.google.com/document/d/18kGXAgeVP1kOlJHZJ2KmV_mkxY61f255LQjPcr6uwnI/edit?pli=1

https://docs.google.com/document/d/13YVWHUXug6XrngQD3a52IgzPxMH2M9sgECkb7peRckA/edit?pli=1

Hope you will have some times to see the problem.