training DDPG agent to drive in unreal engine

Good Morning,

I'm trying to train a ddpg agent to drive a vehicle in a scene built in Unreal Engine. This prevent any parallel computing. There is not any error but I dont know if I'm doing things correctly. The scene is built with OpenstreetMap and RoadRunner, then is loaded in UnrealEngine, then packed and used by simulink trough the "3d Scene configuration block"

I decided to use DDPG beacuse it admits continuous actions, that i strongly needed

Basically I have a set of 41 observations. The simulation runs at 60 fps so my sample time is 1/60. The vehicle sends data to the agent, and the agent performs actions. These actions are the torque request to the electric motor of the vehicle and the wheel angle. Naturally these actions are continuous, not discrete. I am also using the default agent, so the actor and the critic are also default.

The vehicle is on a track and when it touches the lines of the track the simulation ends. Reward is given to the agent through a system of "gates", such that when the vehicle touches one of them it gets a point, these gates are set at contant intervals along the track. I'm not sure whether reward should be given constantly or at certain points like I'm doing now. I tried both ways. The problem is that when I try to train my agent, either it takes too long or it does not converge at all, I dont know which one of the two is actually true. In addition I dont know if i tuned hyperparameters the right way. It keeps doing the wrong things like touching the outer lines or driving backwards, even if it does not get any reward for that

I would also ask if there is any guide on how to tune hyperparameters.

Thanks in advance, any help is highley appreciated.

Answer 1

[Hari]

Hi MARCO BETTIATI,

I understand that you're training a DDPG agent to drive a vehicle in Unreal Engine, facing issues with training duration, convergence, and want to implement reward structuring and hyperparameter tuning.

I assume you are using Unreal Engine integration for simulations. To the trouble shoot the issue you can try the following methods:

1. Reward Structure: Balance continuous and discrete rewards. Continuous rewards for maintaining the vehicle on track and discrete rewards at gates can guide the agent effectively.
2. Hyperparameter Tuning: Begin with default settings. Adjust learning rates and the discount factor as initial steps. Experiment with replay buffer size and batch size for further tuning.
3. Ensure observation and action spaces are accurately defined.
4. Customize actor and critic networks to fit your problem better.
5. Use reward shaping, curriculum learning, or pre-training in simplified environments to enhance learning efficiency.

You can refer to the following documentation for more information.

• DDPG agents in MATLAB: https://www.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html
• Integrating MATLAB/Simulink with Unreal Engine: https://www.mathworks.com/help/driving/unreal-engine-scenario-simulation.html
• Training options and hyperparameter tuning: https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html

Hope this helps!