Open-TeleVision allows VR-type control of remote robot

Teleoperated data collection and learning setup. Left: our teleoperation system. Right: we train an imitation policy for each task with action chunking transformer. The transformer encoder captures the relationship of image and proprioception tokens and the transformer decoder outputs action sequences of a certain chunk size. Credit: arXiv (2024). DOI: 10.48550/arxiv.2407.01512

A combined team of roboticists from MIT and the University of California, San Diego, has developed a new type of remote control for robots called Open-TeleVision that allows for VR-type control. The team has posted a paper describing their robot system on the arXiv preprint server. They also posted videos on YouTube showing the robot and its pilot in action.

AI technology applied to robotics has produced gradual improvements, but to date, there are no autonomous robots that operate as smoothly as a biological creature, including humans. And most in the field do not expect parity any time soon. Some in the field have returned to the idea of remotely controlled robots.

In this new study, the team from MIT and UCSD has made the experience of controlling a remote robot much more immersive by giving the pilot a VR-type experience.

Most VR consumer products are intended for gaming—a player dons a VR helmet/mask and uses control devices to wander around and manipulate objects in a virtual world. The researchers have taken this idea a step forward by using much the same technology to allow a pilot to wander around and manipulate real objects by using the senses and body of a remote robot.

With the Open-TeleVision system, a pilot dons a VR helmet, allowing him or her to see via the robot’s eyes in a stereoscopic manner. The pilot is able to move the robot’s head by turning his/her head, and to manipulate the robot’s arms, hands and fingers by moving his or her own.

Instead of wearing sensors on the arms, hands and fingers, movement of the limbs is captured using remote sensors similar to those in a Kinect system. The system allows a pilot to feel as if they are in a remote location performing activities such as picking up balls and placing them in cups while the robot does the actual work.

The research team demonstrated the utility of the system by allowing a team member at MIT to control a robot at UCSD, while feeling as if they themselves were at the remote site. They suggest, upon further refinement, systems based on Open-TeleVision could be used for applications including remote surgery, search and rescue efforts in dangerous places, or even to explore other planets.

More information:
Xuxin Cheng et al, Open-TeleVision: Teleoperation with Immersive Active Visual Feedback, arXiv (2024). DOI: 10.48550/arxiv.2407.01512

Journal information:
arXiv

© 2024 Science X Network

Citation:
Open-TeleVision allows VR-type control of remote robot (2024, July 15)
retrieved 15 July 2024
from https://techxplore.com/news/2024-07-television-vr-remote-robot.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.