How to deal with both discrete and continuous action in the same environment?
5 views (last 30 days)
Show older comments
Suppose in a custom environment there are two actions- one is discrete and the other continuous. How can I build such an environment in MATLAB? All examples are only of one type -either discrete or continuous.
Thanks.
1 Comment
Answers (1)
Ronit
on 25 Apr 2024
Hi,
To create a custom environment class, “rl.env.MATLABEnvironment” function in MATLAB can be used.
But it appears that an action space combined of both discrete and continuous actions cannot be created. Here’s a workaround for handling the mixed action space within the step function of your environment by manually processing the action input based on your specific requirements.
Following is a sample custom environment with this workaround:
classdef MyCustomEnv < rl.env.MATLABEnvironment
properties
% Define properties for your environment
State = [];
end
methods
function this = MyCustomEnv()
% Initialize Observation settings
ObservationInfo = rlNumericSpec([2 1]);
ObservationInfo.Name = 'My Observations';
ObservationInfo.Description = 'Description of the observation';
% Initialize Action settings
% Here, you might define either a continuous or discrete action space
% and handle the mixed nature internally.
ActionInfo = rlNumericSpec([2 1], 'LowerLimit', [-1; -10], 'UpperLimit', [1; 10]);
ActionInfo.Name = 'My Actions';
% The constructor must call the step function with these info
this = this@rl.env.MATLABEnvironment(ObservationInfo, ActionInfo);
% Initialize environment state
this.State = [0; 0];
end
% Reset function
function InitialObservation = reset(this)
this.State = [0; 0];
InitialObservation = this.State;
end
% Step function
function [Observation, Reward, IsDone, LoggedSignals] = step(this, Action)
LoggedSignals = [];
% Here, you would manually interpret the Action vector
% For example, assuming the first element is discrete (after scaling/rounding)
% and the second is continuous.
DiscreteAction = round(Action(1)); % Simplify for example
ContinuousAction = Action(2);
% Implement your environment dynamics here using the interpreted actions
% Example update of the state
this.State = this.State + [DiscreteAction; ContinuousAction];
Observation = this.State;
Reward = -sum(this.State.^2);
IsDone = norm(this.State) > 10;
end
end
end
This approach requires careful handling and proper description, as it does not follow the standard approach of defining and using the action space. You'll need to ensure that any agents interacting with this environment are correctly configured and generated to match the environment's expected format.
Refer to the documentation of "rl.env.MATLABEnvironment" for more information on how to create and customize your own environment: https://www.mathworks.com/help/reinforcement-learning/ug/create-custom-environment-from-class-template.html
I hope this helps!
0 Comments
See Also
Categories
Find more on Image Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!