MEMS Seminar: "Toward Foundational Robot Manipulation Skills"
Thomas Lord Department of Mechanical Engineering & Materials Science, Fall 2023 Seminar Series with Dieter Fox, Professor of Computer Science & Engineering at the University of Washington
"Toward Foundational Robot Manipulation Skills"
The last years have seen astonishing progress in the capabilities of generative AI techniques, particularly in the areas of language modeling and image generation. Key to the success of these techniques is the availability of very large sets of images and text along with models that are able to digest such large datasets. Unfortunately, we have not been able to replicate the success of generative AI models in the context of robotics. An important problem is the lack of data suitable to train powerful, general models for robot decision making and control.
In this talk, I will discuss our ongoing efforts toward developing the models and generating the kind of data that might lead to foundational manipulation skills for robotics. To generate large amounts of data, we sample many object rearrangement tasks in physically realistic simulation environments and apply task and motion planning to generate high quality solutions for them. We will then train manipulation skills so that they can be used across a broad range of object rearrangement tasks in unknown, real-world environments. We believe that such skills could provide the glue between generative AI reasoning and robust execution in the real world.
DIETER FOX is Senior Director of Robotics Research at NVIDIA and Professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington, where he heads the UW Robotics and State Estimation Lab.