Abstract

Many objects that humans use on a daily basis involve complex interactions that can be difficult to reproduce with robot manipulators. In this work, we represent these interactions using a method to 1) specify object-centric motion trajectories using a sequence of wrenches that are learned, and 2) training an agent to learn to apply these wrenches and track the trajectory in a differentiable simulation environment. Current methods for learning task-oriented dexterous manipulation policies require a large number of environment interactions resulting in the need to learn in simulation. However, this introduces a separate challenge of bridging the sim-to-real gap. To address the challenge of sample complexity, we propose an object-centric task space which defines a trajectory of poses and wrenches to make policy learning more tractable and robust to perturbations in the dynamics by being conditioned on the underlying forces to apply. Our method, \methodName, combines this form of task supervision with a differentiable simulator that enables policy gradients to be computed through the environment dynamics model. We develop a set of dexterous manipulation tasks controlling articulated objects with joints of varying degrees of stiffness to highlight the benefits of our task space and learning algorithm compared to other model-free and model-based methods. Then, we demonstrate how object-centric motion trajectories can be leveraged to learn dexterous motion policies for hands of different morphologies. Lastly, we empirically validate the benefits of our method over a set of model-based and model-free RL baselines.

Paper

@InProceedings{srinivasan2024dexmots,
    title = {DexMOTS: Learning Contact-Rich Dexterous Manipulation in an Object-Centric Task Space with Differentiable Simulation},
    author = {Srinivasan, Krishnan and Heiden, Eric and Ng, Ian and Bohg, Jeannette and Garg, Animesh},
    booktitle = {International Symposium of Robotics Research (ISRR)},
    year = {2024},
}

Video

Demo Videos

Method Diagram

Acknowledgements

This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project. It was adapted to be mobile responsive by Jason Zhang for PHOSA. The code can be found here.