Changes in Predicted values after training

Hello Friends,

I need some help as regards predicted values from an LSTM neural netwrok that I designed. See the attached picture of the main code for the LSTM.

After training , the predicted values are always different from the previous simulation.

In the training options, I tried to set the BiasInitializer to zeros and the WeightsInitializer as well to ones but still the changes in predicted values always occur.

I will be glad if someone can guide me on this point.

[Umar]

Hi Abiodun Abiola,

Firstly, ensure that the random seed is fixed for reproducibility. You can set the random seed at the beginning of your code using rng(seed_number). Additionally, consider normalizing your input data consistently to prevent fluctuations in predictions.Moreover, the randomness in weight initialization might still affect the network's behavior. You can try fixing the random seed for weight initialization using rng(seed_number, 'twister') before creating the network layers. This step can help maintain consistency in weight initialization across simulations.By addressing these aspects, you can enhance the reproducibility of your LSTM network's predicted values and minimize the variations observed between different runs.

Answer 1

[Kaustab Pal]

Hi @Abiodun Abiola,

The output of a neural network (NN) will vary every time you train it. This happens because before training, the NN weights are initialized randomly. As a result, after the training process, the NN will converge to a different local minima in the loss landscape.

To prevent this from happening, I suggest you use the “rng(SEED)” command where “SEED” is a constant integer. When you use this command, the weights are initialized with the same random values every time you train. While this will increase the chances of getting the same output vector every time, it's not guaranteed. There are still other factors that may contribute to different output vectors:

1. Many training algorithms, such as Stochastic Gradient Descent (SGD) and its variants, involve random sampling of data (e.g., mini-batches). Even with a fixed seed, the order in which data is shuffled and sampled can affect the training process.
2. Some operations in neural network libraries (such as parallel computations on GPUs) can be non-deterministic. This means that even with the same initial conditions, the results of certain operations might vary slightly due to the nature of floating-point arithmetic and parallel processing.
3. Techniques such as dropout, batch normalization, and weight decay can introduce stochasticity into the training process. For example, dropout randomly deactivates neurons during training, leading to different network configurations in each epoch.

By setting a random seed and controlling other sources of randomness, you can achieve more consistent results across different runs. However, be aware that some variability may still occur due to the factors mentioned above.

You can read more about setting random seeds using the “rng” command in the documentation here: https://www.mathworks.com/help/matlab/ref/rng.html

Hope this helps.