The RL Agent block only supports Normal and Accelerator simulation modes.

I am working on Reinforcement Learning to control coupled-tank system. I created a simulink environment for Deep Deterministic Policy Gradient (DDPG) agent. When I run the model in normal mode it works well. But when I choose the external mode it gives the below error. Is there any solution to use the DDGP agent in external mode on the Simulink?

The RL Agent block only supports Normal and Accelerator simulation modes.

Answer 1

[Emmanouil Tzorakoleftherakis]

Hi Mehmet,

As I mentioned in the other thread, assuming you want to train your agent in External Model, I believe this is a current limitation as of R2020b. If you want to use a trained agent in External Mode, probably the easiest way is to replace the RL Agent block with a MATLAB Function block that does inference. Inside the MATLAB Fcn block, you can call the function that is generated by generatePolicyFunction.

Hope that helps

[mehmet bozuyla]

Thank you Emmanouil. I will try your suggestion.

[mehmet bozuyla]

Hi Emmanouil.

I tried your suggestion, but I will have another question now. The generatePolicyFunction only takes "observation" attribute as input. But in my DDPG agent, I use "observation", "reward" and "isdone" attributes as input. How can I solve this problem?

Best regards.

[Emmanouil Tzorakoleftherakis]

Trained agents don't need/utilize the IsDone flag and the reward signal anymore. They only need observations as inputs. If it helps, think of the agent as the training algorithm. After you train, you only need to extract and use the policy/neural network in your system, no need to have an agent anymore.

[mehmet bozuyla]

I have used MATLAB R2019b. Is your suggestion valid for this version?

[Emmanouil Tzorakoleftherakis]

External mode for policy inference would require codegen from deep learning objects which was made available in R2020b. In prior releases you can still run inference from a MATLAB Fcn block with 'coder.extrinsic' but you wouldn't be able to generate code from it.

[mehmet bozuyla]

thank you

[Filip Poloczek]

I tried the solution above and generated the function with generatePolicyFunction.

I use the following code for the matlab function:

function y = fcn(u)
output = evaluatePolicy(u)
y = output

If I try now to use the external mode, I get the following error code:

For deep learning, the simulation target language must be set to C++.

Had you the same problems or do you know a solution for that problem?