(Games as) Robotics Simulators

I'm trying to drive a car in a game the hard way--by extracting only that info that could reasonably be obtained by real sensors, and taking only those actions that could really be done by an on-board computer.

At the end, given - game camera - noisy "IMU" data (linear and angular accelerations with added noise) - noisy "GPS" (pose with added noise) - high-level or coarse waypoints (e.g. a path from the Autodrive mod)

We should be able to

• Do neural semantic segmentation and depth estimation, and then create a segmented point cloud at at least 5 Hz.
• Do SLAM on this (or at least extract some obstacles for a local planner).
• Predict our dynamics based on previously-recorded ground-truth trajectory data and control inputs.
• Use this dynamics model to do MPC.
• Feed the MPC control solution back into the simulator as pedals and wheel inputs.

As far as being an simulation environment for training neural networks goes, I see an immediate path forward for learning low-dimensional dynamics models, which is my main interest. However, it might also be possible to use graphics pipleline alterations libraries (my eye is on Crosire Reshade) to get ground-truth data for the perception part as well (depth and segmentation).

Why this and not some purpose-made simulator like CARLA or Nvidia Universe or whatever it's called? Because (1) those simulators are unreasonably hard to install, or have onerous system requirements, and (2) there are some high-level tasks to do in this environment (i.e., it's more fun).

Entries

Reasons 19 April 2023

Data Exfiltration 20 April 2023

GPU Perf+Segmented Point Clouds 16 May, 2023

Things to Do Next 18 May, 2023