The goal of this Ph.D. project is to present selected challenges regarding facial analysis within the fields of Human Biometrics and Human Genetics. In the course of the Ph.D. nine papers have been produced, eight of which have been included in this thesis. Three of the papers focus on face and gender recognition, where in the gender recognition papers the process of human perception of gender is analyzed and used to improve machine learning algorithms. One paper addresses the issues of variability in human annotation of facial landmarks, which most papers regard as a static “gold standard”. However, we document intra- and inter-operator variability associated with annotating these landmarks, which is a valuable result for applications that are sensitive to such variability. One paper presents a comprehensive proof-of-concept study of the prediction of facial characteristics based solely on genetic information, a new area that holds great potential. Two papers explore the connection between minor physical anomalies in the face and schizophrenic disorders. Schizophrenia is a life long disease, but early discovery and treatment can have a significant impact on the course of the disease. Finally, one paper presents a novel appearance model that is a fusion of the active appearance models and the Riemannian elasticity framework.