Post-Doctoral Position: Synthesis Of Videos Driven By Text And Audio

Description of the research project

In recent years, voice interaction with computers has made significant progress. Virtual agents offer a user-friendly human-machine interface while reducing maintenance costs. Speechbased interaction is already effective, as proved by Siri, Google Assistant or Alexa virtual agents, however, their visual counterpart is still far behind. The level of user engagement for audiovisual interactions is much higher than for purely audio interactions. Therefore, it is desirable to be able to associate face visual animations with the generated audio.

A notable advancement in video generation was made by a team at Stanford University in 2019 in partnership with Adobe 1. Their work is aimed at enabling a video editing technology of a person's face-to-face scene to revise its speech script and adapt the rendering automatically based simply on this revised text.

The latest advances in the field of audio-driven face video synthesis were presented in 2. The proposed approach generalizes across different people, to synthesize videos of a target actor with the voice of any actor from an unknown source or even synthetic voices that can be generated using standard text-to-speech approaches.

During this post-doctoral contract, we want to work on the development of a prototype of a Text to Speech to Video technology with a sufficient level of accuracy.

Duration, place to work and supervisors

The funding covers 18 months of post-doc, the desired start is April-May 2022. The postdoctoral fellow will be attached to the LIRIS (Laboratory of Computer Science in Image and Information Systems) on the campus of the University Lyon 2 in Bron. Some stages of work can be conducted in the office of Mon Petit Placement in Lyon.

The post-doc will be supervised by IuliiaTkachenko and Serge Miguet (LIRIS). The project manager on the side of Mon Petit Placement is the technical director of the startup Thibault Jaillon.

References

1 O. Fried, A. Tewari, M. Zollhöfer, A. Finkelstein, E. Shechtman, D. Goldman, K. Genova, Z. Jin, C. Theobalt, M. Agrawala, “Text-based editing of talking-head video”, ACM Transactions on Graphics (TOG), Vol.38, 2019.

2 J. Thies, M. Elgharib, A. Tewari, C. Theobalt, M. Niessner, “Neural Voice Puppetry: Audio-driven Facial Reenactment”, ECCV 2020 (https: // justusthies.github.io/posts/neural-voice-puppetry/ )

Offer Requirements Specific Requirements

The candidate must have a PhD in computer science, specializing in image and video processing

Programming languages: Python/C++

Neural network libraries: PyTorch/Keras/Tensorflow

Programming tools for image analysis: OpenCV

Scientific knowledge: machine learning and deep learning, video analysis and processing

Good writing skills and proficiency in written and spoken English (French is not a requirement)

Contact Information

Organisation/Company: CNRS LIRIS

Organisation Type: Small Medium Enterprise, Start-up

Website: https: // perso.liris.cnrs.fr/itkachenko/

Country: France

应聘时请将简历抄送一份到：chinayanjiusheng@126.com，邮件标题：应聘单位名称、姓名、学历、专业、博士研究生招聘网