Transforming online court hearings with avatars

Since the coronavirus pandemic, video conferences have become an everyday occurrence across a variety of professions. Online court hearings are also becoming increasingly important around the world. In large countries such as Canada and Australia, virtual court hearings have become commonplace to avoid long journeys.

A research team at Fraunhofer Austria is currently working on software to take online court cases to a new level: The aim is to develop virtual avatars that will represent participants in the online courtroom. The participants’ facial expressions and eye movements are recorded by a webcam and then transferred to the avatars. This makes it possible for all participants to make eye contact with each other, simulating face-to-face communication.

Online court hearings are becoming increasingly important. Since the coronavirus crisis, there has been a steady rise in online court cases, with many courts transitioning to online hearings. This offers a number of advantages, for example, hearings can be conducted more quickly, and participants do not need to travel long distances. In addition to streamlining court proceedings, cost and time savings are another benefit of virtual cases.

Critics, however, point out that the required technology is not yet mature and that many procedural questions remain unanswered: How can the quality of the video transmissions be guaranteed? How can hearings be moved to a digital format while respecting the detailed rules of conduct? The researchers at Fraunhofer Austria Research GmbH are working to answer such questions, collaborating with other scientists on this project, including Prof. David Tait, a researcher at Western Sydney University.

They are creating a software-based virtual court system that can be used to optimize and develop video technology applications in civil justice. The plan is to use the virtual courtroom setting initially for misdemeanors, for example in cases involving disputes between neighbors.

In many video conference applications, VR glasses are used to give users the feeling of being in the same physical location; however, the Fraunhofer researchers are not relying on VR in this project. Instead, the video output is displayed on a monitor. The participants communicate in the virtual courtroom using avatars—3D graphical characters that represent real people.

Before the start of the hearing in the virtual courtroom, the participants select the appropriate role specification for them, such as judge, defense attorney, prosecutor, witness or defendant. A webcam records their face. Used in conjunction with a 3D graphics engine, the software is equipped with eye tracking to detect the direction in which the user looks. This eye movement is translated into the avatar’s head movement, simulating direct eye contact between the interlocutors. It is this eye contact that is currently missing in video conferences via MS Teams or Zoom.

Video conferences on an equal footing

“Our virtual court system does not require any complicated technical equipment. Participants can be located anywhere—in a public building, police station or at home—as long as they have access to a laptop and webcam. That is all they need to take part in a virtual court hearing,” says Volker Settgast, senior researcher at the Visual Computing business unit at Fraunhofer Austria in Graz. In the virtual representation of the courtroom, all users can see each other and tell where everyone is looking thanks to eye tracking. The webcam detects not only eye movements but also facial expressions.

“As the webcam records mouth movements and facial expressions, conclusions can be made about the emotional state of participants,” explains Settgast. “In traditional video conferences, the video image is transmitted, but this is not necessary with our system thanks to the use of avatars. Only the audio stream and the data obtained from eye tracking and facial expression recognition are transmitted. The data stream to be transmitted is therefore reduced.”

There are plans to adapt the virtual courtrooms to meet country-specific requirements and work is also underway to improve the avatar animation. For example, Settgast and his team want to incorporate participants’ hands with gesture recognition. A browser-based version of the software is also planned for the future. Beta testing has already begun and the first workshops and user tests took place at Harvard University’s Visualization Research and Teaching Laboratory, Department of Earth and Planetary Sciences, and at the University of Montreal’s Cyberjustice Laboratory.

“Ultimately, we want to move the court setting to a completely virtual environment and enable civil court video conferences to be held on an equal footing. As eye movements, facial expressions and gestures are visible, online hearings will have the same personal dynamic as face-to-face interactions in the future,” says the researcher.