Unreal Engine Simulation for Robots

Robotics System Toolbox™ provides a co-simulation framework that models driving algorithms in Simulink^® and visualizes their performance in a virtual simulation environment. This environment uses the Unreal Engine^® from Epic Games^®.

Note

Simulating models in the 3D visualization environment requires Simulink 3D Animation™.

Simulink blocks related to the simulation environment can be found in the Robotics System Toolbox > Simulation 3D block library. These blocks provide the ability to:

Configure prebuilt scenes in the simulation environment.
Place and actuate robots within these scenes.
Set up camera and lidar sensors on the vehicles.
Simulate sensor outputs based on the environment around the robot.
Obtain ground truth data for semantic segmentation and depth information.

This simulation tool is commonly used to supplement real data when developing, testing, and verifying the performance of robotics algorithms. In conjunction with a robot model, you can use these blocks to perform realistic closed-loop simulations that encompass the entire robot control stack, from perception to control.

For more details on the simulation environment, see How Unreal Engine Simulation for Robots Works.

Unreal Engine Simulation Blocks

To access the Robotics System Toolbox > Simulation 3D library, at the MATLAB^® command prompt, enter robotsim3dlib.

Scenes

To configure a model to co-simulate with the simulation environment, add a Simulation 3D Scene Configuration block to the model. Using this block, you can choose from a prebuilt scene where you can test and visualize your driving algorithms.

The toolbox includes these scenes.

Scene	Description
Open Surface	Flat, black pavement surface with no road objects
Offroad pit mining scene	Offroad pit mining scene containing an excavation area and haul truck roads

Robots and Vehicles

To define a virtual robot or vehicle in a scene, add one of these blocks to your model:

Simulation 3D Robot — Specify a robot rigid body tree model from the Robotics System Toolbox Robot Library Data or import from any rigid body tree model. Actuate the robot by setting joint configurations of the rigid body tree, and enable movement by specifying translations and rotations of the robot base at each timestep.
Simulation 3D Physics Dump Truck — Load a dump truck model. Control the movement of the dump truck by specifying steering angles and velocities at each timestep. Actuate the lift by specifying an optional input signal for lift angles at each timestep.

Sensors

You can define virtual sensors and attach them at various positions on the vehicles. The toolbox includes these sensor modeling and configuration blocks.

Block	Description
Simulation 3D Camera	Camera model with lens. Includes parameters for image size, focal length, distortion, and skew.
Simulation 3D Lidar	Scanning lidar sensor model. Includes parameters for detection range, resolution, and fields of view.

Algorithm Testing and Visualization

Robotics System Toolbox simulation blocks provide the tools for testing and visualizing path planning, robot control, and perception algorithms.

Path Planning and Vehicle Control

You can use the Unreal Engine simulation environment to visualize the motion of a vehicle in a prebuilt scene. This environment provides you with a way to analyze the performance of path planning and vehicle control algorithms. After designing these algorithms in Simulink, you can use the robotsim3dlib library to visualize vehicle motion in one of the prebuilt scenes.

Perception

Robotics System Toolbox provides several blocks for detailed camera and lidar sensor modeling. By mounting these sensors on robots within the virtual environment, you can generate synthetic sensor data or sensor detections to test the performance of your sensor models against perception algorithms.

Closed-Loop Systems

After you design and test a perception system within the simulation environment, you can then use it to drive a control system that actually steers a vehicle. In this case, rather than manually set up a trajectory, the robot uses the perception system to fly itself. By combining perception and control into a closed-loop system in the 3D simulation environment, you can develop and test more complex algorithms, such as intelligent bin picking, and pit mining.