Modify Reinforcement Learning Algorithm

Hi all,

I am looking for way to change the actor's objective function according to a paper (https://cs-people.bu.edu/rmancuso/files/papers/ICRA21_1616_FI.pdf).

I need to change it by adding two terms to the policy optimizatiton function in an attempt to stabilise my agent's actions.

Where can I find the base SAC algorithm file? How can I minimise the use of custom code in order to approach this problem?

Thanks.

[xiang]

Hello, have you solved your problem? I have the same question

Answer 1

[Shivansh]

Hi Tech,

I understand that you want to change the actor's objective function for a RL model according to CAPS research paper and find the base SAC algorithm for implementation.

To implement the changes to the actor's objective function based on the CAPS paper in MATLAB, you will need to modify the Soft Actor-Critic (SAC) algorithm. Here are the steps you can follow:

1. Find the base SAC algorithm file: The SAC algorithm is commonly implemented in deep reinforcement learning libraries. You can search for SAC implementations in these libraries on GitHub or other code repositories. Look for the main SAC algorithm file that contains the actor's objective function.
2. Minimize the use of custom code: Since you want to minimize the use of custom code, you can consider using the Deep Learning Toolbox in MATLAB. This toolbox provides built-in functions and tools for deep reinforcement learning, including neural network training and optimization.
3. Adapt the SAC algorithm in MATLAB: Once you have found the base SAC algorithm file, you can adapt it to MATLAB by rewriting the code using the Deep Learning Toolbox. This may involve translating the code from the original library to MATLAB syntax and using the corresponding functions in the Deep Learning Toolbox.
4. Add the CAPS regularization terms: To stabilize the agent's actions, you need to add the two CAPS regularization terms to the actor's objective function. You can find the specific equations for the temporal smoothness and spatial smoothness terms in the CAPS paper. Implement these terms as additional components in the actor's objective function.
5. Train and evaluate the modified SAC algorithm: Once you have made the necessary modifications and added the CAPS regularization terms, you can train and evaluate the modified SAC algorithm using your desired environment and tasks.

Additionally, you may need to adjust hyperparameters and experiment with different settings to achieve the desired stabilization effect. For more information on Deep Learning Toolbox, you can refer to the following link: https://www.mathworks.com/help/deeplearning/index.html?searchHighlight=deep%20learning%20toolbox&s_tid=srchtitle_support_results_1_deep%20learning%20toolbox.