>during the training process especially at the beginning of my training I >get extremely high training error after that this error starts go down >slowly. After approximately 500 Epochs the training error comes near to >zero (e.g. 0.006604). Then, I took the final obtained model to measure >its accuracy against the testing data, I've got about 89.50%. Does that >normal?
That is not unusual.
>I mean getting a high training error rate at the very beginning of my >training process.
Yes. It's not unusual
>Another thing, I'd like to mention is that I've noticed that every time i >decrease the number of the hidden nodes the results become better at the >end of my training.
This, also is not unusual. It often occurs when an overfit net (i.e., see below for H > Hub and H >> Hub) is overtrained.
[ I N ] = size(input)
[ O N ] = size(target)
[ O N ] = size(output)
Ntrn = 0.7*N
Ntrneq = Ntrn*O
H = numberofhiddennodes
[ H I ] = size(IW)
[ H 1 ] = size(b1)
[ O H ] = size(LW)
[ 1 H ] = size(B2)
Then, the number of unknown weights is
Nw = (I+1)*H + (H+1)*O
The number of unknowns exceeds the number of equations when
Nw > Ntrneq
H > Hub
where the upper bound is
Hub = (Ntrneq-O)/(I+O+1)
When H > Hub there are two common ways to mitigate this.
1. STOPPED TRAINING: Stop training when the error on a validation subset
increases for a specified number of epochs. The default in the NN Toolbox is
a. Automatic trn/val/tst subset data division in the ratios 0.7/0.15/0.15
b. A 6 epoch limit of continuous val subset error increases.
2. REGULARIZATION: Using a trn/val/tst subset data division in the ratio
0.85/0.0/0.15 with the training function TRAINBR that uses a "regularized"
error function consisting of a weighted sum of sum-squared-error and
3. Other useful search terms in the NEWSGROUP, ANSWERS and the internet are
Hope this helps.
Thank you for formally accepting my answer