Orbiting astronaut oversees robot team on Earth

Astronaut Frank Rubio aboard the International Space Station collaborated with a small team of robots on Earth to accomplish a complex task—a first test of a new approach to combine human and robotic capabilities for our return to the moon and beyond.

The two-hour telerobotics test took place in a simulated planetary environment in the German Aerospace Center, DLR, at Oberpfaffenhofen near Munich. Working from Europe’s Columbus module, NASA astronaut Frank Rubio directed a trio of robotic assets to remove a seismometer from a lunar lander and place it on the ground to detect moonquakes.

Frank served as overseer for the robot team, working on a “point and click” basis. He took direct control as needed using a force-feedback device. A DLR robotic lander arm was combined with DLR’s Rollin’ Justin humanoid robot and ESA’s four-wheeled, two-armed Interact rover.

The goal is to shed light on what astronauts need when operating robots with varying levels of independence, known as “scalable autonomy.” This approach enables astronauts to have more control over the robots.

In the test scenario, the first in the “Surface Avatar” series, a stuck pin had to be removed from the lander before the seismometer could be positioned by Justin. Meanwhile the Interact rover had been damaged by a quake during a surface sampling operation, so that the sample still stuck in its gripper needed to be retrieved by Justin and be placed on the lander.

In the event, Frank accomplished all the tasks in time, while also giving the team valuable feedback on features of the user interface and the robots. Those will be implemented and validated in the next session of Surface Avatar in January 2024.

This first test opens the way to a follow-up Surface Avatar test with a larger team of robots, planned to be undertaken by Danish ESA astronaut Andreas Mogensen during his Huginn mission later this year.

The distances involved in spaceflight make it impractical to perform direct remote control of robots on the moon or other planetary surfaces from Earth. Instead, the idea is to have astronauts in orbit control these robots—to explore and take action on the moon, then later other alien environments, without the cost and risk of landing.

“Our METERON project undertaken with DLR, proved the feasibility of this force-feedback control technique from local orbit, expanding to work with astronauts on the ISS and culminating in an outdoor campaign in the moon-like environment of Mount Etna in Sicily,” explains Thomas Krueger, head of ESA’s Human Robot Interaction Lab, partnering with DLR’s Institute of Robotics and Mechatronics.

“With Surface Avatar we combining direct teleoperation approach with supervised autonomy to control multiple robotic assets to accomplish complex tasks: picture a building site on Earth, where a crane might work alongside a bulldozer or excavator to get a big task done.

“What we want to find out with this testing is how the astronauts perform in practice. What control mode do they choose? Do they have situational awareness by seeing the world through the eyes of multiple robots? And what judgements they make in their head about how to act?”

The Surface Avatar test scenario assumes the relatively structured environment of a planetary surface base, where the robots already know how to interact with some of the objects around them and the other robots in place. So, an augmented reality display produces a list of options for the controller to select.

• 

• 

If the desired option is not available, then the astronaut switches to direct control, feeling precisely what the robot arm experiences through a force-feedback control device, allowing a high level of precision.

“We are excited to be a step closer to giving astronauts and experts on Earth a wide range of possibilities to command and manage teams of different robots in space,” says Principal Investigator Neal Y. Lii from the DLR Institute of Robotics and Mechatronics. “We will be able to use our robots on the surface as their physical avatars and intelligent co-workers to execute ever more complex tasks.”

Thomas concurs, “This human-robot collaboration paves the way for future missions and permanent outposts on the moon and beyond.”