Study urges reappraisal of alleged racial ‘bias’ in face recognition

A team of biometrics researchers have looked closely at the factors behind recent incidents that led to face recognition technology being labelled “biased” or “racist”, and found that image quality and variability – rather than racism inherent in datasets or the biometric technology itself – could be the main reason for any disparity.A team from Florida Institute of Technology and University of Notre Dame , including Krishnapriya KS, Kushal Vangara , Michael C. King, Vítor Albiero and Kevin Bowyer, launched a methodical investigation into differences in face recognition accuracy between African-American and Caucasian image cohorts of the MORPH dataset.The team noted that for all four matchers considered, the impostor and the genuine distributions were statistically significantly different between cohorts. For a fixed decision threshold, the African-American image cohort indeed has a higher false match rate and a lower false non-match rate.However, they also noted that ROC curves compare verification rates at the same false match rate, but the different cohorts achieve the same false match rate at different thresholds. This means that ROC comparisons are not relevant to operational scenarios that use a fixed decision threshold.Ultimately, the team found that using ICAO compliance as a standard of image quality, that the initial image cohorts have unequal rates of good quality images. “The ICAO-compliant subsets of the original image cohorts show improved accuracy, with the main effect being to reducing the low-similarity tail of the genuine distributions”. “Based on the face detection rates and the failure-to-enroll rates in Section 3, we find no good evidence for a difference in the face detection or failure-to-enroll rate between the African-American and Caucasian cohorts. “”Demographic balance was not a design goal the for VGGFace dataset [13], used to train VGG, or for the VGGFace2 dataset [15], used to train ResNet. Based on manual inspection of VGGFace2, we estimate the ratio of Caucasian to African-American subjects in the VGGFace2 dataset as in the range of 5:1 to 6:1.”Yet the d-prime values for the ResNet impostor and genuine distributions show that inherent face recognition accuracy is at least as good for the African-American cohort as for the Caucasian cohort. ResNet's better ROC or higher d-prime for the AfricanAmerican cohort was not achieved through a demographically-balanced training dataset, demonstrating that, at least in this instance, balanced training data is not a requirement to obtain balanced, or better, accuracy”.

Also in the news:

Rio to trial face recognition at carnival Australia to place biometrics at centre of digital ID programme US fingerprinting refugee families on Nauru