Create policy evaluation function for RL agent

7 views (last 30 days)
Hello,
I'm using the Reinforcement Learning Toolbox to train an agent for a particular task. Once the training is complete, I want to generate a policy evaluation function so I can use it in a simulink file. I can create the policy function using "generatePolicyFuncion" command. The matlab script that is generated is the following:
function action1 = evaluatePolicy(observation1)
%#codegen
% Reinforcement Learning Toolbox
% Generated on: 23-Sep-2019 09:27:39
action1 = localEvaluate(observation1);
end
%% Local Functions
function action1 = localEvaluate(observation1)
persistent policy
if isempty(policy)
policy = coder.loadDeepLearningNetwork('agentData.mat','policy');
end
action1 = predict(policy,observation1);
end
When I run the simulink file, the following errors pop up:
Simulink does not have enough information to determine output sizes for this block. If you think the errors below are inaccurate, try specifying types for the block inputs and/or sizes for the block outputs.
Component:MATLAB Function | Category:Coder error
Dot indexing is not supported for variables of this type.
Function 'MATLAB Function' (#171.286.341), line 13, column 11:
"coder.loadDeepLearningNetwork('agentData.mat','policy')"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.
Function 'MATLAB Function' (#171.365.371), line 15, column 19:
"policy"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.
Function 'MATLAB Function' (#171.140.167), line 7, column 11:
"localEvaluate(observation1)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'test_RL/MATLAB Function'
Component:MATLAB Function | Category:Coder error
Simulink cannot determine sizes and/or types of the outputs for block 'test_RL/MATLAB Function' due to errors in the block body, or limitations of the underlying analysis. The errors might be inaccurate. Fix the indicated errors, or explicitly specify sizes and/or types for all block outputs.
Component:MATLAB Function | Category:Coder error
Dot indexing is not supported for variables of this type.
Function 'MATLAB Function' (#171.286.341), line 13, column 11:
"coder.loadDeepLearningNetwork('agentData.mat','policy')"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.
Function 'MATLAB Function' (#171.365.371), line 15, column 19:
"policy"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Function call failed.
Function 'MATLAB Function' (#171.140.167), line 7, column 11:
"localEvaluate(observation1)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'test_RL/MATLAB Function'
How can I solve this issue?
Thanks

Accepted Answer

Emmanouil Tzorakoleftherakis
Edited: Emmanouil Tzorakoleftherakis on 25 Sep 2019
Can you try defining the size of inputs and outputs in the MATLAB Function block? This seems to be coming up a lot in the error message
  3 Comments
J
J on 17 Oct 2019
Hi there,
Were you able to resolve this issuse? I am facing the same problem as you and haven't been able to figure it out...
SomeMatlabUser
SomeMatlabUser on 20 Feb 2020
Edited: SomeMatlabUser on 20 Feb 2020
Any Update? I am having the same problem. Also there is no documentation by Mathworks itself how to setup an Simulinkmodel executing the policy for code generation.

Sign in to comment.

More Answers (1)

Sviatoslav Klos
Sviatoslav Klos on 3 Mar 2020
Hello.
I experienced the same issue. If you do not need code generation (only normal mode simulation is requested) I might suggest to substitute your two functions with the following code and insert it as a MATLAB User-defined fcn. Please, respond if it suits you.
function action1 = evaluatePolicy(observation1)%, policy) %#codegen
coder.extrinsic('evalin', 'predict');
action1 = single(0);
policy = evalin('base', 'policy') ;
action1 = predict(policy,observation1);
end
  2 Comments
Juan Nunez
Juan Nunez on 15 May 2020
Hello Sviatoslav,
Thnk your for your response.
I'm sorry for my late reply. I tried your suggestion, but the following error keeps popping up:
"Variable 'action1' has conflicting type definitions. To resolve, verify that all preceding definitions of 'action1' are consistent.
Function 'MATLAB Function' (#420.371.409), line 14, column 1:
"action1 = predict(policy,observation1)"
Launch diagnostic report.
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'Copy_of_RF_FOC6/MATLAB Function'
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'Copy_of_RF_FOC6/MATLAB Function'"
Sviatoslav Klos
Sviatoslav Klos on 25 May 2020
I think this is your project specific error. You might check the action of the RL agent. I might suggest to test this function with some RL example from MATLAB and the try to go into your application with obtained conclusions.

Sign in to comment.

Products


Release

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!