Using Reinforcement Learning algorithm to optimize parameter(s) of a controller
21 views (last 30 days)
Show older comments
HazwanDrK
on 15 Jul 2020
Edited: Mehrdad Moradi
on 26 Jul 2021
Hello,
First of all, I'm relatively new to reinforcement learning. I have a project in which I need to utilize RL to fine tune one, if not many parameters of an already well built controller, in my case it's a discrete controller. I have came across many papers describing the use of RL specifically for control but not many on optimization which I have trouble in understanding the concept. Perhaps, can someone shed some knowledge on the use of RL for parameters fine tuning, like how is it different than the RL controller concept and how is it going to run in parallel with the controller. I'm more than happy if you can share with me the references, if any. Thanks!
0 Comments
Accepted Answer
Emmanouil Tzorakoleftherakis
on 16 Jul 2020
Hi Hazwan,
The main difference between using RL for control vs parameter tuning is that in the first case the policy will directly output, e.g. torque, or whatever is your control input. In the latter case, the output of the policy would be parameter values, e.g., if you are trying to tune a PID, the policy would output 3 numbers, Kp, Ki and Kd. Obviously the observations/inputs to the policy as well as the reward would probably need to be different too.
To your question on how the latter could run in parallel with the controller, I can see two scenarios:
1) Using RL for finding static gains. In this case you train, you get the constant parameter values the RL policy finds, and then you discard the policy and adjust your controller gains with these numbers
2) Using RL for finding dynamic/observation-based parameters. This would be in some sense similar to gain scheduling and for this case you would run the policy in parallel with the controller. The idea would be the same(i.e. the policy would output parameter values) but it would do so all the time, thus updating the controller parameters dynamically based on observations.
Hope that helps.
8 Comments
Emmanouil Tzorakoleftherakis
on 22 Jul 2020
I would say you need the error yes if you are planning on tracking a collection of step responses. Otherwise, you would be basically overfitting to a single reference value if that makes sense.
You don't need a separate function for the reward, it can be incorporated in the step function as shown in the link you mentioned.
More Answers (1)
Mehrdad Moradi
on 26 Jul 2021
Edited: Mehrdad Moradi
on 26 Jul 2021
But how configure RL to find a static gains while action signal a a time series of different values? Is there any guideline about it?
0 Comments
See Also
Categories
Find more on Environments in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!