Update matrix values inside MATLAB Function block

Hi all!

I'm having trouble with updating a matrix inside a MATLAB function block. I'm trying to implement a Q-learning algorithm using only basic MATLAB and Simulink for vehicle control. When I try to update my Q-table, which I defined as a parameter in the MATLAB Function workspace, it gives me the following error:

'Expected either a logical, char, int, fi, single, or double. Found an mxArray. MxArrays are returned from calls to the MATLAB interpreter and are not supported inside expressions. They may only be used on the right-hand side of assignments and as arguments to extrinsic functions.'

My outside Simulink initialization code is:

%tuneable parameters
epsilon = 0.9;
epsilonDecayRate = 0.001;
epsilonMin = 0.01;

%sampleTime = 0.1;
gamma = 0.95;
Pedal = 10; %#
Steer = 30; %deg

%discretizing state and action space
discVx = [5 6 7 8 9 10 11 12 13 14 15];
discVy = [-5 -4.5 -4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0];
discR = linspace(0,1,11);

discPed = linspace(0,1,Pedal+1);
discSteer = linspace(-200,100,(300/Steer)+1);

%skeleton arrays for search
actInfo = [];
for i=1:(Pedal+1)
    for j=1:((300/Steer)+1)
        actInfo(end+1,:) = [discPed(i) discSteer(j)];
    end
end
numofacts = (Pedal+1)*((300/Steer)+1);

obsInfo = [];
for i=1:length(discVx)
    for j=1:length(discVy)
        for k=1:length(discR)
            obsInfo(end+1,:) = [discVx(i) discVy(j) discR(k)];
        end
    end
end
numofstats = length(discVx)*length(discVy)*length(discR);

%Q-table
Q = zeros([numofstats numofacts]);
Q = randi([-5000 -3000],[numofstats numofacts]);

And the MATLAB Function is:

function [pedal,steer] = trainMyQtable(v_x, v_y, r, v_x_, v_y_, r_,...
    reward, Q, obsInfo, actInfo, epsilon, epsilonDecayRate, epsilonMin,...
    discVx, discVy, discR, lastPed, lastSteer, gamma)

%discretize and detect states
[dump,index1] = min(abs(discVx-v_x_));
[dump,index2] = min(abs(discVy-v_y_));
[dump,index3] = min(abs(discR-r_));
laststate = [discVx(index1);discVy(index2);discR(index3)];
[dump,index1] = min(abs(discVx-v_x));
[dump,index2] = min(abs(discVy-v_y));
[dump,index3] = min(abs(discR-r));
newstate = [discVx(index1);discVy(index2);discR(index3)];
lastObsIndx = ismember(obsInfo, laststate);
newObsIndx = ismember(obsInfo, newstate);

%update table value
actIndx = ismember(actInfo, [lastPed lastSteer]);
TD_error = reward + gamma * max(Q(newObsIndx,:)) - Q(lastObsIndx,actIndx);
Q(lastObsIndx,actIndx) = Q(lastObsIndx,actIndx) + alpha*TD_error;

%get action with exploration
choice = rand(1);
if choice > epsilon
    pedal = randsample(actInfo(:,1),1);
    steer = randsample(actInfo(:,2),1);
else
    [~,newActIndx] = max(Q(newObsIndx,:));
    [pedal,steer] = actInfo(newActIndx,:);
end
epsilon = max([epsilon*epsilonDecayRate epsilonMin]);

The problem is with the line

Q(lastObsIndx,actIndx) = Q(lastObsIndx,actIndx) + alpha*TD_error;

Can anyone give me a way to get around this issue? Or maybe a more clever way to implement a Q-table? Any insight would be appreciated. Thank you!

[Mitchell Thurston]

you might be able to declare Q as a fi type? I don't know enough about Q learning to say how you should design the table, but since the error says fi type is allowed, you could just declare Q like

Q = randi([-5000 -3000],[numofstats numofacts], 1, 32, 23);

Not sure what level of precision you need, but the "32, 23" gives single precision.

https://www.mathworks.com/help/fixedpoint/ref/embedded.fi.html

[Szilard Hunor Toth]

Thank you for the suggestion! I tried all of these types the error mentions, but it persists. The Simulink says

Expected either a logical, char, int, fi, single, or double. Found an mxArray. MxArrays are returned from calls to the MATLAB interpreter and are not supported inside expressions. They may only be used on the right-hand side of assignments and as arguments to extrinsic functions. Function 'Agent/MATLAB Function' (#33.855.869), line 20, column 51: "alpha*TD_error"

Though in the model manager, the variable is recognized as a fi type, so there shouldn't be a problem.

Answer 1

[Andy Bartlett]

Something to be aware of is that MATLAB Function Block primarily uses code generation mode and not interpreted mode.

Interpreted Mode

Let's call running code in "regular" MATLAB usage as "interpreted mode."

In interpreted mode, you can think of all numeric variables as being represented as MxArrays.

MxArray are effectively a structure that contains or points to meta data like type, dimensions, complexity, etc.

The MxArray structure also provides pointers to access stored values that live elsewhere in memory.

As an example let's consider

myVar = uint16([4, 5, 6])

The MxArray representing myVar contains meta data like, type is 'uint8', complexity is real, dimensions 1 by 3, etc.

The MxArray also provides access to the 6 bytes that encode myVar's three values 4, 5, and 6.

Code Gen Mode

When generating code for MATLAB code and/or a Simulink model for simulation or production code generation, the variables will be stripped down to "leanest essence."

Normally, MxArray's are no longer to be found. The outer structure and all the meta data make inefficient in the context of code generation.

When generating C code, the example above, myVar, would appear in code as just the following.

uint16_T myVar[3] = {4U, 5U, 6U}

The meta data will not directly appear in the generated code, but will appear indirectly as needed. For example, myVar was declared as type uint16_T with dimension 3.

Same Meaning But Clearly Different

Representing myVar as an MxArray for interpreted mode as a structure that points to meta data and raw data is clearly different from representing myVar in C code as raw numeric type value. In both cases, myVar represents the same thing and the data values like 3, 4, and 5 should be the same. But clearly, their representations are not directly compatible.

The error from MATLAB Function Blocks is indicating that an attempt is being made to mix these incompatible representations. MATLAB Function Block, like MATLAB Coder, attempts to do everything in code generation mode. Somehow a MxArray is coming into play and causing an incompatibility that code generation wants to be plain C/C++.

Consider Debuging with MATLAB Coder

If you have MATLAB Coder available, it might be helpful to debug the issue using MATLAB Coder.

Put your function trainMyQtable in a standalone M-file.

Setup a call to codegen to define the input arguments.

Call codegen and use the created code gen report to narrow down the source of the issue.

For my own work, I find that approach helpful 90% of the time.