TY - JOUR

T1 - Multilinear Modelling of Faces and Expressions

AU - Grasshof, Stella

AU - Ackermann, Hanno

AU - Brandt, Sami Sebastian

AU - Ostermann, Jörn

N1 - Funding Information: This article was supported by German Research Foundation Grant DFG RO 2497/12-2.

PY - 2021/10

Y1 - 2021/10

N2 - In this work, we present a new versatile 3D multilinear statistical face model, based on a tensor factorisation of 3D face scans, that decomposes the shapes into person and expression subspaces. Investigation of the expression subspace reveals an inherent low-dimensional substructure, and further, a star-shaped structure. This is due to two novel findings. (1) Increasing the strength of one emotion approximately forms a linear trajectory in the subspace. (2) All these trajectories intersect at a single point - not at the neutral expression as assumed by almost all prior works - but at an apathetic expression. We utilise these structural findings by reparameterising the expression subspace by the fourth-order moment tensor centred at the point of apathy. We propose a 3D face reconstruction method from single or multiple 2D projections by assuming an uncalibrated projective camera model. The non-linearity caused by the perspective projection can be neatly included into the model. The proposed algorithm separates person and expression subspaces convincingly, and enables flexible, natural modelling of expressions for a wide variety of human faces. Applying the method on independent faces showed that morphing between different persons and expressions can be performed without strong deformations.

AB - In this work, we present a new versatile 3D multilinear statistical face model, based on a tensor factorisation of 3D face scans, that decomposes the shapes into person and expression subspaces. Investigation of the expression subspace reveals an inherent low-dimensional substructure, and further, a star-shaped structure. This is due to two novel findings. (1) Increasing the strength of one emotion approximately forms a linear trajectory in the subspace. (2) All these trajectories intersect at a single point - not at the neutral expression as assumed by almost all prior works - but at an apathetic expression. We utilise these structural findings by reparameterising the expression subspace by the fourth-order moment tensor centred at the point of apathy. We propose a 3D face reconstruction method from single or multiple 2D projections by assuming an uncalibrated projective camera model. The non-linearity caused by the perspective projection can be neatly included into the model. The proposed algorithm separates person and expression subspaces convincingly, and enables flexible, natural modelling of expressions for a wide variety of human faces. Applying the method on independent faces showed that morphing between different persons and expressions can be performed without strong deformations.

KW - 3D-reconstruction

KW - expression transfer

KW - HOSVD

KW - person transfer

KW - Statistical shape model

KW - tensor model

UR - http://www.scopus.com/inward/record.url?scp=85114612751&partnerID=8YFLogxK

U2 - 10.1109/TPAMI.2020.2986496

DO - 10.1109/TPAMI.2020.2986496

M3 - Article

VL - 43

SP - 3540

EP - 3554

JO - IEEE Trans. Pattern Anal. Mach. Intell.

JF - IEEE Trans. Pattern Anal. Mach. Intell.

IS - 10

M1 - 9067086

ER -