System Identification fitness criterion (NRMSE vs NMSE)

Hi guys !

I have a fundamental question regarding the fitness criterion when performing a model estimation for system identification. This far, I have obtained a good model for a driving simulator identification task. In this post, I want to discuss the SISO vertical model. The training data has been obtained by an acceleration sensor which delivers the vertical acceleration signal. Measurements have been performed in a frequency range of 0.2-20Hz. It is obvious that for small frequencies the sensor noise is considerable.

After estimating the (nonlinear) system models, I have calculated the goodnessOfFit using the NRMSE and NMSE focus. I have found that the NMSE yields significantly better fit results compared to the NRMSE. I have understood that the NRMSE is the better choice for describing a 1:1 fit, either the model fits perfectly or it doesn't. However, the NMSE seems to punish outliners less harsh than the NRMSE does.

Thus, I want to argument that my model is good based upon a NMSE measure. The reason being is that for the lower frequency parts the identified model is robust against noise and does not model it. Therefore, it is obvious that the model will not represent the noise in the training signal and is better described by a NMSE criterion than a NRMSE one.

The following plot shows what i mean. Training was done using a local linear model approach with a concatenation of single sines as training data. The model was cross-validated using sweep and noise test signals:

Original training signal and model fit:

To put the plot in numbers:

Fit using NRMSE for the training signal : 87%

Fit using NMSE for the training signal : 98%

Similar values for the test/cross-validation signals.

So my final question is: What do you think of this argumentation. From the plot it is clear that the obtained model is sufficient and good enough to fully represent the system dynamics. I would like to pursue my argumentation using the NMSE as a measure of fit instead using the NRMSE. Though, I have found that in most literature the NRMSE was used most often, however being performed on more 'clinical' training data without too much of a noise.

Thank you very much !

Answer 1

[Rajiv Singh]

A good model should have goodness of fit measure less than 1, indicating that the error (measured_data - model_reponse) is smaller than what you would get if just fit a constant to the data (measured_data - mean(measured_data)). Both NMSE and NRMSE work on this measure:

r = norm((measured_data-model_response)) / norm((measured_data-mean(measured_data)))

We want r<1. For models that produce r < 1, NMSE = r^2 will be more forgiving since r^2 < r and a value close to zero is better. Note that NRMSE =r.

Note that the compare function reports (1-NRMSE)*100 as fitness measure.