Face2Face: Real-time Face Capture and Reenactment of RGB Videos

In a groundbreaking achievement, researchers have developed an innovative approach for real-time facial reenactment of monocular target video sequences. The system, dubbed Face2Face, uses a novel technique to animate the facial expressions of a source actor and re-render the manipulated output video in a photorealistic fashion.

At its core, Face2Face is designed to track facial expressions of both the source and target videos using a dense photometric consistency measure. This allows for seamless blending of the re-targeted face with the real-world illumination, creating an uncanny sense of realism.

The process begins by capturing a monocular video stream from a commodity webcam, which serves as the source sequence. The system then uses non-rigid model-based bundling to address the under-constrained problem of facial identity recovery from monocular video. This enables the accurate retrieval and warping of the mouth interior that best matches the re-targeted expression.

Reenactment is achieved through fast and efficient deformation transfer between the source and target sequences. At run time, the system tracks facial expressions using a dense photometric consistency measure, ensuring that the re-targeted face blends seamlessly with the real-world illumination.

To demonstrate the capabilities of Face2Face, researchers have set up a live setup where YouTube videos are reenacted in real-time. The results are nothing short of astonishing, with the synthesized target face convincingly rendered on top of the corresponding video stream.

The implications of Face2Face are far-reaching, with potential applications in fields such as entertainment, education, and even therapy. Imagine being able to relive a cherished memory or experience through the eyes of someone else – Face2Face makes this a reality.

In recognition of its groundbreaking achievements, Face2Face has received numerous accolades, including the Outstanding Doctoral Dissertation Award and Best Student Project in the Electronic Theater at SIGGRAPH 2016. With its potential for widespread impact, Face2Face is poised to revolutionize the way we interact with video content and experience the world around us.

Face2facergb