AI revolutionizes training for sensorimotor skills from surgery to tennis

The new wave of powerful generative artificial intelligence (AI) systems such as ChatGPT and DALL-E has introduced powerful capabilities such as writing and image generation. In recent studies, researchers have developed new AI technologies that can be used to transfer human expertise in hands-on, complex skills through data-driven, trained technological models—from a beginner being coached on a tennis serve to a medical intern learning how to perform complicated surgeries.

Texas A&M University Professor Dr. Alfredo Garcia, Ph.D. student Chenliang Li, and a team of collaborators at the University of Minnesota have developed new methodologies necessary to create such an advanced AI model. These new methodologies have been published in the Proceedings of Neural Information Processing Systems as well as in the Operations Research Journal.

“Human sensorimotor expertise is achieved when someone can perform a task consistently and correctly even when conditions are not the same,” Garcia said. “With the right model for both experts and novices, we can use AI models to transfer that knowledge in a scalable fashion—so someone can slow down and perfect a skill step-by-step.”

The researchers’ newly developed methodologies help evaluate the actions of AI and provide rewards when tasks are completed successfully by estimating the structure of Markov Decision Processes (MDPs) based on provided data. This allows researchers to create advanced AI models of human expertise, such as a model of an expert surgeon performing movements and adjustments during surgery in response to changing conditions.

For example, take the ability to perform a complex surgery. With Garcia’s model, AI would learn from a surgeon repeatedly demonstrating a successful surgery. An AI model could be trained based on the surgeon’s actions.

In a scenario where a medical student is practicing in a virtual environment or with a doll, the model can be used to show how the operation is going compared to the actions of the expert. AI would evaluate the success of current steps using MDP’s reward system to acknowledge successful completion or failure.

“The AI model can serve as a coach providing feedback to the student in training,” Garcia said.

“I like to think of it as a serve in tennis—it can be such a difficult skill to learn, especially if you learn it when you’re older,” Garcia said. “But if you have a coach, they can look at you and give you very specific directions on ways to improve because they understand the inner dynamics of the movement. We want to replicate what a tennis coach does, but purely based on data.”

The proposed AI system can potentially be used to teach skills that humans typically take years to develop on their own, which may usher in a new era of experts into fields such as medicine, manufacturing and even athletics.

“All these tasks that require you to push, to press, to cut—they use your senses and your ability to perceive things around you,” Garcia said. “Machines don’t know what that’s like. It’s all a very embodied experience and we want to build models that account for this aspect.”