Faced with two independent studies that found its facial recognition software returns inaccurate or biased results, Amazon has repeatedly claimed that the researchers failed to use the software, called Rekognition, in the way the company has instructed police to use it.
However, the only law enforcement agency Amazon has acknowledged as a client says it also does not use Rekognition in the way Amazon claims it recommends, Gizmodo has learned. In doing so, the law enforcement agency undermines the very argument Amazon uses to discredit critical research about Rekognition.
Rekognition is a software suite which purports to serve various functions ranging from identifying specific facial features to comparing similarities in a large volume of photos. This gives the software potential value to law enforcement customers seeking, for example, to match a mugshot to security cam footage of a robbery.
Critics are concerned that automating the policing process could have dire consequences if the software displays racial or other biases, especially as Amazon has pitched Immigration and Customs Enforcement on using this product. So far Amazon’s frontline defence against advocates and academics has been: Researchers are setting the search parameters incorrectly.
The parameter in question is something Amazon refers to as a “confidence threshold” or “similarity threshold.” When acting on a query, the software also informs the user how closely the two photos match, scored out of 100 per cent, with the option to set a threshold that filters results below the defined percentage.
When an ACLU argued that the researchers simply failed to follow best practices. The company claimed in a statement:
While 80% confidence is an acceptable threshold for photos of hot dogs, chairs, animals, or other social media use cases, it wouldn’t be appropriate for identifying individuals with a reasonable level of certainty. When using facial recognition for law enforcement activities, we guide customers to set a higher threshold of at least 95% or higher.
Last week, researchers from the MIT Media Lab published a study indicating that Rekognition’s facial analysis function showed it struggled to correctly identify women of colour. Once again, Amazon suggested the results stemmed not from bias in the software itself, but from incorrect threshold settings:
We are not yet aware of the threshold used in this research, but as you will see below, the results are much different when run with the recommended confidence level. […] when using facial recognition to identify persons of interest in an investigation, law enforcement should use our recommended 99% confidence threshold (as documented), and only use those predictions as one element of the investigation (not the sole determinant).
Currently, the only known law enforcement client using Rekognition is the Washington County Sheriff’s Office in Oregon. (Other clients may exist, but Amazon has refused to disclose who they might be.) Worryingly, when asked by Gizmodo if they adhere to Amazon’s guidelines where this strict confidence threshold is concerned, the WCSO Public Information Officer (PIO) replied, “We do not set nor do we utilise a confidence threshold.”
Amazon’s documentation states that law enforcement clients may “use a similarity threshold lower than 99% for scenarios that benefit from a larger set of potential matches [such as] finding missing persons,” but according to a source with knowledge of the WCSO’s Rekognition setup and usage who asked to remain anonymous for fear of retribution, the software is deployed in cases ranging from theft to homicide.
Part of a slide presentation created by WCSO — disclosed in a Freedom of Information Act request filed by the ACLU — clearly shows a correctly identified image only earning a 96.03-per cent confidence score, a result which might have been omitted had Amazon’s guidance been followed to the letter.
To be clear, the failure to abide by Amazon’s precise suggestions does not appear to be the result of negligence on the WCSO’s part—quite the opposite. “The sheriff’s office has never viewed Rekognition search results as ‘matches.’ The technology provides leads for investigators, but ultimately identifying suspects is a human-based decision-making process, not a computer-based one,” the PIO later clarified. “Notably, and per policy, our investigators are not permitted to take enforcement action based solely on a potential identification lead.”
The same source within WCSO told Gizmodo that a “confidence score is not used anywhere in the process to make a decision and so no threshold is used in the search.”
The PIO further informed Gizmodo that, while Amazon did supply documentation and other support on the software end, no direct training was given to the investigators who continue to use the suite. One Systems Project Analyst working in nearby Clackamas County — whose correspondence was made public via the ACLU-obtained FOIA documents — wrote that Rekognition’s “documentation is very lacking or wrong.”
While the company’s agreement with WCSO states that the “customer will provide AWS with timely and appropriate access to personnel, documentation, data, systems, and facilities,” Amazon declined to state whether it tracks the confidence thresholds employed by its clients.
A spokesperson pointed us to an Amazon Web Services blog post from last week that states that the company will “suspend people’s use of our services if we find they’re using them irresponsibly or to infringe on people’s civil rights.”
An Amazon spokesperson clarified that law enforcement clients failure to use a 99-per cent confidence threshold does not constitute an irresponsible application of Rekognition.
Currently, there is no evidence that WCSO or other law enforcement clients, if they exist, have employed Rekognition for abusive purposes. Still, the excuses Amazon has made for their software and the actual experience of its use do not align.
Amazon has, in its public-facing blog posts, supported legislation that would guide the use of software like Rekognition. “It is a very reasonable idea, however, for the government to weigh in and specify what temperature (or confidence levels) it wants law enforcement agencies to meet to assist in their public safety work,” Dr. Matt Wood, Amazon Web Service’s general manager of deep learning, wrote in a July post.
In the opinion of those Congressional lawmakers who have repeatedly sought information to potentially regulate the use of facial recognition, however, Amazon “has failed to provide sufficient answers” when pressed in July and November.
“I think this demonstrates the ridiculousness of Amazon’s position,” Matt Cagle, a technology and civil liberties attorney with the ACLU of Northern California, told Gizmodo, “on the one hand they talk about guidelines, but then the company’s own customers may not actually be following them, and that is unacceptable when we’re talking about a technology as dangerous as face surveillance that is being used by law enforcement agencies. Amazon should act. It should heed the calls of Congress, of the public, and even its own workers and get out of the surveillance business.”