The Racist History Behind Facial Recognition

Researchers recently learned that Immigration and Customs Enforcement used facial recognition on millions of driver’s license photographs without the license-holders’ knowledge, the latest revelation about governments employing the technology in ways that threaten civil liberties.

But the surveillance potential of facial recognition — its ability to create a “perpetual lineup” — isn’t the only cause for concern. The technological frontiers being explored by questionable researchers and unscrupulous start-ups recall the discredited pseudosciences of physiognomy and phrenology, which purport to use facial structure and head shape to assess character and mental capacity.

Artificial intelligence and modern computing are giving new life and a veneer of objectivity to these debunked theories, which were once used to legitimize slavery and perpetuate Nazi race “science.” Those who wish to spread essentialist theories of racial hierarchy are paying attention. In one blog, for example, a contemporary white nationalist claimed that “physiognomy is real” and “needs to come back as a legitimate field of scientific inquiry.”

More broadly, new applications of facial recognition — not just in academic research, but also in commercial products that try to guess emotions from facial expressions — echo the same biological essentialism behind physiognomy. Apparently, we still haven’t learned that faces do not contain some deeper truth about the people they belong to.

Composite photographs, new and old

One of the pioneers of 19th-century facial analysis, Francis Galton, was a prominent British eugenicist. He superimposed images of men convicted of crimes, attempting to find through “pictorial statistics” the essence of the criminal face.

Galton was disappointed with the results: He was unable to discern a criminal “type” from his composite photographs. This is because physiognomy is junk science — criminality is written neither in one’s genes nor on one’s face. He also tried to use composite portraits to determine the ideal “type” of each race, and his research was cited by Hans F.K. Günther, a Nazi eugenicist who wrote a book that was required reading in German schools during the Third Reich.

Galton’s tools and ideas have proved surprisingly durable, and modern researchers are again contemplating whether criminality can be read from one’s face. In a much-contested 2016 paper, researchers at a Chinese university claimed they had trained an algorithm to distinguish criminal from noncriminal portraits, and that “lip curvature, eye inner corner distance, and the so-called nose-mouth angle” could help tell them apart. The paper includes “average faces” of criminals and noncriminals reminiscent of Galton’s composite portraits.

The paper echoes many of the fallacies in Galton’s research: that people convicted of crimes are representative of those who commit them (the justice system exhibits profound bias), that the concept of inborn “criminality” is sound (life circumstances drastically shape one’s likelihood of committing a crime) and that facial appearance is a reliable predictor of character.

It’s true that humans tend to agree on what a threatening face looks like. But Alexander Todorov, a psychologist at Princeton, writes in his book “Face Value” that the relationship between a face and our sense that it is threatening (or friendly) is “between appearance and impressions, not between appearance and character.” The temptation to think we can read something deeper from these visual stereotypes is misguided — but persistent.

In 2017, the Stanford professor Michal Kosinski was an author of a study claiming to have invented an A.I. “gaydar” that could, when presented with pictures of gay and straight men, determine which ones were gay with 81 percent accuracy. (He told The Guardianthat facial recognition might be used in the future to predict I.Q. as well.)

The paper speculates about whether differences in facial structure between gay and straight men might result from underexposure to male hormones, but neglects a simpler explanation, wrote Blaise Agüera y Arcas and Margaret Mitchell, A.I. researchers at Google, and Dr. Todorov in a Medium article. The research relied on images from dating websites. It’s likely that gay and straight people present themselves differently on these sites, from hairstyle to the degree they are tanned to the angle they take their selfies, the critics said. But the paper focuses on ideas reminiscent of the discredited theory of sexual inversion, which maintains that homosexuality is an inborn “reversal” of gender characteristics — gay men with female qualities, for example.

“Using scientific language and measurement doesn’t prevent a researcher from conducting flawed experiments and drawing wrong conclusions — especially when they confirm preconceptions,” the critics wrote in another post.

Echoes of the past

Parallels between the modern technology and historical applications abound. A 1902 phrenology book showed how to distinguish a “genuine husband” from an “unreliable” one based on the shape of his head; today, an Israeli start-up called Faception uses machine learning to score facial images using personality types like “academic researcher,” “brand promoter,” “terrorist” and “pedophile.”

Faception’s marketing materials are almost comical in their reduction of personalities to eight stereotypes, but the company appears to have customers, indicating an interest in “legitimizing this type of A.I. system,” said Clare Garvie, a facial recognition researcher at Georgetown Law.

“In some ways, they’re laughable,” she said. “In other ways, the very part that makes them laughable is what makes them so concerning.”

In the early 20th century, Katherine M.H. Blackford advocated using physical appearance to select among job applicants. She favored analyzing photographs over interviews to reveal character, Dr. Todorov writes. Today, the company HireVue sells technology that uses A.I. to analyze videos of job applicants; the platform scores them on measures like “personal stability” and “conscientiousness and responsibility.”

Cesare Lombroso, a prominent 19th-century Italian physiognomist, proposed separating children that he judged to be intellectually inferior, based on face and body measurements, from their “better-endowed companions.” Today, facial recognition programs are being piloted at American universities and Chinese schools to monitor students’ emotions and engagement. This is problematic for myriad reasons: Studies have shown no correlation between student engagement and actual learning, and teachers are more likely to see black students’ faces as angry, bias that might creep into an automated system.