Deepfakes now come with a realistic heartbeat, making them harder to unmask

Imagine a world where deepfakes have become so good that no detection mechanism can unmask them as impostors. This would be a bonanza for criminals and malignant state actors: for example, these might use deepfakes to slander rival political candidates or frame inconvenient defenders of human rights.

This nightmare scenario isn’t real yet, but for years, methods for creating deepfakes have been locked in a “technological arms race” against detection algorithms. And now, scientists have shown that deepfakes have gained a significant advantage: the lack of a pulse no longer gives them away.

“Here we show for the first time that recent high-quality deepfake videos can feature a realistic heartbeat and minute changes in the color of the face, which makes them much harder to detect,” said Dr. Peter Eisert, a professor at Humboldt University of Berlin, and the corresponding author of a new study in Frontiers in Imaging.

Deepfake creators use deep learning to manipulate videos and audio files. They alter facial expressions and gestures, for example, swapping these between different people. Their purpose isn’t necessarily malign: for example, apps that can turn you into a cat or digitally age you are immensely popular and harmless fun.

The analysis of the transmission of light through the skin and underlying blood vessels has long been indispensable in medicine, for example in pulse oximeters. Its digital cousin, so-called remote photoplethysmography (rPPP), is an emerging method in telehealthcare, which uses webcams to estimate vital signs. But rPPP can, in theory, also be used in deepfake detectors.

In recent years, such experimental rPPP-based deepfake detectors have proven good at distinguishing between real and deepfaked videos. These successes led some experts to judge that current deepfakes couldn’t yet mimic a realistic heart rate. But now, it appears that this complacent view is outdated.

Fake it till you make it

Eisert and colleagues first coded a state-of-the-art deepfake detector which automatically extracts and analyzes the pulse rate from videos. It uses novel methods to compensate for movement and remove noise, and needs an input video of the face of a single person in just 10 seconds to work.

The authors also created their own dataset of driving videos, used to create deepfakes of different target identities with the facial motion of the captured videos. During filming, an ECG tracked the heartbeat of the protagonists, which then allowed the researchers to confirm that rPPP measurements made by their new detector were highly accurate.

There was a difference of only two to three beats per minute between estimates and the true pulse rate. For good measure, the authors also let their detector loose on two older, widely used collections of videos of real persons. Here, too, they were able to extract heartbeat signals from all genuine videos.

But what would happen if they used the same detector to analyze known deepfakes?

To test this, Eisert and colleagues used recent deepfake methods to swap faces between genuine videos in their collection. To their surprise, their detector perceived a pulse in the deepfakes as well—even though they hadn’t consciously put one in. This nonexistent pulse typically appeared highly realistic.

Taking heart

“Our results show that a realistic heartbeat may be added by an attacker on purpose, but can also be ‘inherited’ inadvertently from the driving genuine video. Small variations in skin tone of the real person get transferred to the deepfake together with facial motion, so that the original pulse is replicated in the fake video,” said Eisert.

Fortunately, there is reason for optimism, concluded the authors. Deepfake detectors might catch up with deepfakes again if they were to focus on local blood flow within the face, rather than on the global pulse rate.

“Our experiments have shown that current deepfakes may show a realistic heartbeat, but do not show physiologically realistic variations in blood flow across space and time within the face,” said Eisert.

“We suggest that this weakness of state-of-the-art deepfakes should be exploited by the next generation of deep fake detectors.”