Beyond Teloperation: Learning from Diverse Human and Simulation Data

Introduction

Recent progress in training foundation models for robotics has been driven largely by large, real-world robot datasets. Collecting such data requires labor-intensive and expensive teleoperation to provide high-quality expert demonstrations, making it intractable to scale across diverse tasks and environments. As a result, robotics datasets still remain much smaller compared to the massive corpora that drove revolutions in NLP and CV. By contrast, off-domain data sources such as human videos and large-scale simulation exist in abundance, and could unlock robot learning at scale.

However, both human and simulation data are mismatched for robot learning. Human demonstrations are fundamentally different in embodiment from robots, preventing direct motion transfer, whereas simulations, despite their scalability, often lack the fidelity needed to faithfully model the real world. This workshop asks the pivotal question: How can we move beyond teleoperation and effectively use abundant off-domain data sources for robot learning?

In the Beyond Teloperation: Learning from Diverse Human and Simulation Data workshop, we are particularly interested in methods that

• extract actionable, transferable representations from human videos for robot policy learning
• develop world models and real-to-sim-to-real frameworks that ground simulation in real-world human activity
• combine small, high-quality teleoperation datasets with large-scale off-domain sources to balance fidelity and scalability
• bridge embodiment and sim-to-real gaps to ensure generalization across domains

We aim to convene a diverse set of perspectives—from robot learning, 3D computer vision, embodied AI, simulation, and human-robot interaction—to discuss what it means to scale robot learning beyond teleoperation, and to define the research agenda for making these methods robust and deployable in real-world settings.