Accéder directement au contenu Accéder directement à la navigation
Nouvelle interface
Communication dans un congrès

Instruction-driven history-aware policies for robotic manipulations

Abstract : In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.
Type de document :
Communication dans un congrès
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-03775734
Contributeur : Pierre-Louis Guhur Connectez-vous pour contacter le contributeur
Soumis le : mardi 13 septembre 2022 - 07:35:59
Dernière modification le : vendredi 4 novembre 2022 - 15:02:48

Fichier

2209.04899.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-03775734, version 1

Collections

Citation

Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia, Makarand Tapaswi, Ivan Laptev, et al.. Instruction-driven history-aware policies for robotic manipulations. CoRL 2022 - Conference on Robot Learning, Dec 2022, Aukland, New Zealand. ⟨hal-03775734⟩

Partager

Métriques

Consultations de la notice

12

Téléchargements de fichiers

6