some of the saved agents in DQN reinforcement learning algorithm do not reproduce the training rewards

Hello everyone,

I am training a DQN agent in my reinforcement learning problem in Simulink Matlab 2019a. I have set a save criteria of agents with episode rewards equal to 6. My problem is that after a number of episodes, some agents will save based on my save criteria, but when I use them, some of the agents do not reproduce rewards equal to 6 or more but the amount of rewards is less than 6. If anyone could help me I will appriciate.

[Madhav Thakker]

Hi masoud,

Do you save the agent as soon as you get an episode with reward 6? Are you using an epsilon-greedy policy to train the agents? How less are the rewards when you use the saved agent?

[masoud k]

Hi Madhav,

The criteria for saving the candidate agents is getting reward of 6 or more of an episode.

I use epsilon-greedy policy to train agents with epsilon equal to 0.6 and the decay rate of 0.001.

Some of saved agents come up with an episode reward around 1 and some other with 6 or more.

Answer 1

[Emmanouil Tzorakoleftherakis]

This is a common misconception. When you train an agent and you get a certain reward out of a specific episode, if you were to stop training and run the same episode again with the same agent you would most likely get a different response from the agent and thus a different episode reward.

There are a few reasons for that:

1) During training, the agent explores different options based on some probabilistic exploration strategy. For DQN this strategy is related to the epsilon parameter in DQN options. After training, the agent does not explore anymore and only relies on the underlying neural network for inference, so it makes sense that results are different.

2) It is best practice to randomize various elements of the environment during training to get a more robust policy. In that case there will also be differences in how the environment behaves during and after training, so agent will respond differently as well.

3) Some agents are stochastic. This by itself implies different decisions/behavior even if everything else in the environment remains deterministic.

Hope that helps

[masoud k]

Thank you Emmanouil for your inclusive response.

Just for clarification, for your first reason, we could say that it is likely that the candidate agents do not have the same strategy as they have in the training process. If it is so, how can we make sure that we have got a reliable agent that gets the desirable reward through simulation?

Thank you in advance

[Emmanouil Tzorakoleftherakis]

Typically, exploration slows down over time and the agent starts exploiting more. For example, you can see how epsilon decays for DQN here. If you start getting consistently good rewards at the later stage of training where exploration is low, then that's a good sign that this behavior will carry over after training. You may still not get exactly the same rewards (especially if the minimum epsilon value is nonzero) but you should overall get desirable performance.