MATLAB Answers

RL Agent does not learn

17 views (last 30 days)
Hello,
I'm up to Reinforcement Learning with the RL Toolbox. After I built a custom environment in Simulink I have problems training the PG Agent. The RL problem is to control a system with diffuse irradiance, direct irradiance, and temperature as states and a mass flow rate as actions which can be 0 or 30. While Simulink generates a cost function it is also the reward signal and the objective is to reduce the cost over one episode of 24 time steps.
The code is the following:
obsInfo = rlNumericSpec([3 1]);
obsInfo.Name = 'Observation';
actInfo = rlFiniteSetSpec([0 30]);
actInfo.Elements = [0 30];
actInfo.Name = 'Action';
env = rlSimulinkEnv(mdl,[mdl '/RL Agent'],obsInfo,actInfo);
% Create deep neural network approximator for the actor
net = [ imageInputLayer([3 1 1], 'Normalization', 'none', 'Name', 'state')
fullyConnectedLayer(32, 'Name', 'fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(32,'Name','fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(32,'Name','fc3')
reluLayer('Name','relu3')
fullyConnectedLayer(2, 'Name', 'fc4')
softmaxLayer('Name', 'actionProb') ];
% Create actor
actorOpts = rlRepresentationOptions('LearnRate',0.01,'GradientThreshold',1);
actor = rlStochasticActorRepresentation(net, obsInfo, actInfo, 'Observation', 'state',actorOpts);
% Create Agent
opt = rlPGAgentOptions('DiscountFactor',0.0001);
agent = rlPGAgent(actor);
When I train the agent for 2000 Episodes with different configurations of the neural net I have the problem that it does not converge to a path at all. At some point the policy finds configurations which result in better reward but afterwards the agent does not converge further and does not follow the improved policy.
It would be great if you could help me solve the problem. Do you think this happens due to a insufficient reward signal or does the structure of my neural net not fit my observation and action signals? I also tried using tanhLayer or different amounts of nodes without any success.
Thank you very much for your help!
Best regards
Janika

  0 Comments

Sign in to comment.

Accepted Answer

Emmanouil Tzorakoleftherakis
Edited: Emmanouil Tzorakoleftherakis on 2 Jul 2020
Hello,
It is really hard to say just by looking at the training plot. The first thing I would try is 1) a different agent (maybe DQN since you have a discrete action space). If the agent is still not learning, assuming your network structure is roughly similar to what you see in other shipping examples, it probably has something to do with the reward or lack of exploration. It sounds like this is a controls problem, so I would highly recommend to look at some of the shipping examples to get a feeling of how different agents behave.

  2 Comments

Janika Ofterdinger
Janika Ofterdinger on 5 Jul 2020
Thanks for answering my question!
I already tried different agents such as DQN and the training plot looks similar but the issue may be caused by my reward signal. Due to algebraic loops I had to implement a unit delay which affects the reward and holds it for one sample time delay.
To circumvent the issue of exploration I already set the Entropy Loss Weight of the PG Agent to 1.
I also was wondering if the agent is capable to detect a better policy even if the reward is only a little bit better than the old one. Do you think it is better to shape the reward function in the direction to increase the range between a penalty and a reward?
Thanks again for your support!
Best
Janika
Emmanouil Tzorakoleftherakis
It all depends on the application. On your side you need to make sure that the different terms in the reward signal are comparable and that their relative relationship guides the agent to the desired behavior.
For now, I would probably leave the default values for exploration and focus on the reward. If you start seeing some improvement but the agent is stuck on a local solution, that's a good indication to start playing with exploration parameters.
Also, keep in mind that PG agent is monte carlo-based, so it will likely need more training to learn.

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!