The title says it all; 3D Face Reconstruction from a Single Image! A whole lot of #gamedev is reconstructing characters and worlds that can’t be found IRL. That’s the whole point of an art medium whose biggest draw is escapism. Fantasy art assets will still need to be created by “hand.” But, for the times when you need to create a photo-realistic face, there’s no better way than photogrammetry! And there’s no better, more economical way to do that than by using one image and pushing one (or two) buttons. The best solution, therefor, seems to be 3D Face Reconstruction from a Single Image! This web-app moves #gamedev ever closer to one-button art asset creation!
Of course the aforementioned caveat regarding fantasy art applies, as does a caveat about optimization. The resulting .obj is a mess of a mesh; it will need to be reconstructed as “polygons” “by hand”. Or will it? Elsewhere I’ve extolled ZRemesher’s abilities to automatically retopologize character faces. But 3D Face Reconstruction’s messy meshes may be beyond ZRemesher’s means. I’ll attempt to find out and report back!
From the whitepaper’s abstract:
“3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the availability of multiple facial images (sometimes from the same subject) as input, and must address a number of methodological challenges such as establishing dense correspondences across large facial poses, expressions, and non-uniform illumination. In general these methods require complex and inefficient pipelines for model building and fitting. In this work, we propose to address many of these limitations by training a Convolutional Neural Network (CNN) on an appropriate dataset consisting of 2D images and 3D facial models or scans. Our CNN works with just a single 2D facial image, does not require accurate alignment nor establishes dense correspondence between images, works for arbitrary facial poses and expressions, and can be used to reconstruct the whole 3D facial geometry (including the non-visible parts of the face) bypassing the construction (during training) and fitting (during testing) of a 3D Morphable Model. We achieve this via a simple CNN architecture that performs direct regression of a volumetric representation of the 3D facial geometry from a single 2D image. We also demonstrate how the related task of facial landmark localization can be incorporated into the proposed framework and help improve reconstruction quality, especially for the cases of large poses and facial expressions.”
TL; DR? Then check this out!:
— Reza Zadeh (@Reza_Zadeh) September 17, 2017
And finally, at the risk of repeating what’s embedded in the tweet above, you can get the code here (get it before the web-app crashes)!: