Add scalar inputs to the actor network

I have a CNN based PPO actor critic, and it is working fine, but now I am trying to add three scalar values to the actor network. The observation originally was [12 12 4] as we have a 12*12 environment and 4-channel image:

% Define observation specifications.
obsSize = [12 12 4];
oinfo = rlNumericSpec(obsSize);
oinfo.Name = "observations";

I am trying to add four scalar inputs to the actor network, I used the following actor network, but when I am use the following line of code in training, I have a trouble:

actor(idx) = rlContinuousGaussianActor(actorNetwork,oinfo,ainfo);

Error using rl.internal.validate.mapFunctionObservationInput
Number of input layers for deep neural network must equal to number of observation specifications.

Error in rlContinuousGaussianActor (line 95)
modelInputMap = rl.internal.validate.mapFunctionObservationInput(model,observationInfo,nameValueArgs.ObservationInputNames);

I am not sure how I can change the observation size that I can able to add 3 scalar values. Here are some information about the workspace:

% actorNetwork = 
% LayerGraph with properties:
% Layers: [21×1 nnet.cnn.layer.Layer]
% Connections: [20×2 table]
% InputNames: {'featureinput_3'  'featureinput_1'  'featureinput_2'  'imageinput'}
% OutputNames: {1×0 cell}
% 
% oinfo = 
% rlNumericSpec with properties:
% LowerLimit: -Inf
% UpperLimit: Inf
% Name: "observations"
% Description: [0×0 string]
% Dimension: [12 12 4]
% DataType: "double"
% 
% ainfo = 
% rlNumericSpec with properties:
% LowerLimit: -1
% UpperLimit: 1
% Name: "actions"
% Description: [0×0 string]
% Dimension: [1 2]
% DataType: "double"

[ali farid]

Thank you for your guidance. I revised the actor network, but it seems that the inputs are empty. I put constant inputs w1,w2, w3 using

featureInputLayer(w1,"Name","scalarInput1")

featureInputLayer(w2,"Name","scalarInput1")

featureInputLayer(w3,"Name","scalarInput1")

but still there is an error:

Error using rl.internal.validate.mapFunctionObservationInput

Unable to automatically specify deep neural network observation input layer names because some specifications have similar dimension. Specify "ObservationInputNames" name-value pair when creating function object.

Error in rlContinuousGaussianActor (line 95)

modelInputMap = rl.internal.validate.mapFunctionObservationInput(model,observationInfo,nameValueArgs.ObservationInputNames);

Here is the revised code:

global w1,
global w2, 
global w3;
W=[w1,w2,w3];
w1=2;
w2=1;
w3=2;
 
obsMat = [4 3; 5 3; 6 3; 7 3; 8 3; 9 3; 5 11;  6 11; 7 11; 8 11; 6 12; 7 12; 10 12; ];
sA0 = [2 5];
sB0 = [11 5];
sC0 = [3 2];
 
s0 = [sA0; sB0; sC0];
 
Ts = 0.1;
Tf = 100;
maxsteps = ceil(Tf/Ts);
mdl = "rlA";
open_system(mdl)
 
 
% Define observation specifications.
scalarObs1Info = rlNumericSpec([1 1]);
scalarObs1Info.Name ="scalarObservation1";
scalarObs2Info = rlNumericSpec([1 1]);
scalarObs2Info.Name ="scalarObservation2";
scalarObs3Info = rlNumericSpec([1 1]);
scalarObs3Info.Name ="scalarObservation3";
obsSize = [12 12 4];
oinfo = rlNumericSpec(obsSize);
oinfo.Name = "observations";
allObsInfo = [ scalarObs1Info, scalarObs2Info, scalarObs3Info, oinfo];
allObsInfo(1).Name = "observations";
allObsInfo(2).Name = "scalarObservation1";
allObsInfo(3).Name = "scalarObservation2";
allObsInfo(4).Name = "scalarObservation3";
ActionInfo = rlNumericSpec([1, 2], 'Lowerlimit', -1, 'Upperlimit', 1); ainfo = ActionInfo;
ainfo.Name = "actions";
actInfo.UpperLimit=1;
actInfo.Lowerlimit=-1;
        blks = mdl + ["/Agent A","/Agent B","/Agent C"];
 

env = rlSimulinkEnv(mdl,blks,{allObsInfo,allObsInfo,allObsInfo},{ainfo,ainfo,ainfo});
env.ResetFcn = @(in) resetMap(in, obsMat);
rng(0)
for idx = 1:3
 
lgraph = layerGraph();
    tempLayers = [
    featureInputLayer(w1,"Name","scalarInput1")
    reluLayer("Name","relu_3")
    fullyConnectedLayer(10,"Name","fc_4")];
lgraph = addLayers(lgraph,tempLayers);
 
tempLayers = [
    featureInputLayer(w2,"Name","scalarInput2")
    reluLayer("Name","relu_2")
    fullyConnectedLayer(10,"Name","fc_3")];
lgraph = addLayers(lgraph,tempLayers);
 
tempLayers = [
    featureInputLayer(w3,"Name","scalarInput3")
    reluLayer("Name","relu_1")
    fullyConnectedLayer(10,"Name","fc_2")];
lgraph = addLayers(lgraph,tempLayers);
 
tempLayers = [ 
        imageInputLayer(obsSize,Normalization="none")
        convolution2dLayer(8,16, ...
            Stride=1,Padding=1,WeightsInitializer="he")
        reluLayer
        convolution2dLayer(4,8, ...
            Stride=1,Padding="same",WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(256,WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(128,WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(2,"Name","fc_1")];
lgraph = addLayers(lgraph,tempLayers);
 
tempLayers = [
    concatenationLayer(2,4,"Name","concat")
    softmaxLayer("Name","softmax")];
lgraph = addLayers(lgraph,tempLayers);
 
% clean up helper variable
clear tempLayers;
 
lgraph = connectLayers(lgraph,"fc_2","concat/in3");
lgraph = connectLayers(lgraph,"fc_3","concat/in2");
lgraph = connectLayers(lgraph,"fc_1","concat/in4");
lgraph = connectLayers(lgraph,"fc_4","concat/in1");
 
plot(lgraph);
actorNetwork=lgraph;
actorOptions = rlOptimizerOptions('LearnRate',0.1,'GradientThreshold',inf);  
actor(idx) = rlContinuousGaussianActor(actorNetwork,allObsInfo,ainfo);
 
%Critic network  

Answer 1

[Hassaan]

To add scalar values to the actor network in a reinforcement learning setup using MATLAB, you need to modify both your observation space and your neural network architecture. The error you're encountering is because the number of input layers in your neural network does not match the number of observation specifications.

In your case, you have an original observation space of size [12 12 4] (which is a 4-channel image of size 12x12) and you want to add three additional scalar values. Here's how you can approach this:

Step 1: Modify Observation Specification

You need to create additional observation specifications for your scalar values. Since you have three scalar values, you should create three rlNumericSpec objects with a dimension of [1 1].

Step 2: Modify Neural Network Architecture

Your neural network should have an additional input layer for each of the scalar observations. If your original network has an input layer for the [12 12 4] observation, you need to add three more input layers, each corresponding to one of the scalar values.

Create Observation Specifications for Scalars: Create three rlNumericSpec objects for the scalar values.

scalarObs1Info = rlNumericSpec([1 1]);
scalarObs1Info.Name = "scalarObservation1";

scalarObs2Info = rlNumericSpec([1 1]);
scalarObs2Info.Name = "scalarObservation2";

scalarObs3Info = rlNumericSpec([1 1]);
scalarObs3Info.Name = "scalarObservation3";

Combine All Observation Specifications: Combine your original observation spec (oinfo) with the new scalar observation specs.

allObsInfo = [oinfo, scalarObs1Info, scalarObs2Info, scalarObs3Info];

Modify Your Neural Network: Add three input layers to your existing network, each for one scalar observation. You should name these input layers accordingly (e.g., 'scalarInput1', 'scalarInput2', 'scalarInput3').

Assuming actorNetwork is your existing network, you might need to modify it as follows (this is an example, you might need to adjust based on your network architecture):

layers = [
    imageInputLayer([12 12 4], 'Normalization', 'none', 'Name', 'imageinput')
    %... (your existing layers)
    featureInputLayer(1, 'Name', 'scalarInput1')
    featureInputLayer(1, 'Name', 'scalarInput2')
    featureInputLayer(1, 'Name', 'scalarInput3')
    %... (rest of your network)
];
actorNetwork = layerGraph(layers);

Create the Actor: When creating the actor, use the updated network and observation specification.

actor = rlContinuousGaussianActor(actorNetwork, allObsInfo, ainfo);

This should resolve the issue, as now the number of input layers in your network will match the number of observation specifications. Ensure that the rest of your network architecture is appropriately designed to handle these additional inputs.

------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

Professional Interests

• Technical Services and Consulting
• Embedded Systems | Firmware Developement | Simulations
• Electrical and Electronics Engineering

Feel free to contact me.