NARX - a strange thing happens

I control a lower limb exoskeleton for performing the sit-to-stand exercise using electromiographical signals with a NARX net.

With the standard procedure, I set some of the input samples for training, for validation and for test, starting with open loop and then closing the loop.

Casually, using "net.divideFcn = 'divideind'" I set all input samples for training without validation and test.; Obiously, starting with the same random numbers the final performance is better without validation and test than in the other case, as can be seen in the first two figures obtained in the maglev example.

Then I saved the net parameters for the case of no validation and test and with those parameters I started directly the training in closed loop with identical samples for training, validation and test, as before. As can be seen in the third figure the performace is close to the training without validation and test.

Conclusion: the net parameters for the case of not validation and test are the best also when validation and test are performed.

What is wrong in my reasoning?

Giuseppe

Answer 1

[Krishna]

Hello Giuseppe,

The issue you're encountering is known as overfitting in machine learning. Essentially, by using all available data for training the networks, the model may fit the data very well, possibly too well. Consequently, when testing the model with the same data it was trained on, it will likely perform well because it's already familiar with that data. However, when presented with new data, the model's performance will likely be worse than before because instead of learning general patterns, it has simply memorized the dataset, resulting in overfitting. That is why it is recommended to have 3 sets of dataset training, validation, and testing.

Please go through this documentation to know the importance of these 3 datasets,

https://www.geeksforgeeks.org/training-vs-testing-vs-validation-sets/

Hope this helps.