It looks like Amazon is working on a new Alexa-powered gadget that can listen to you and decide how you feel, and make recommendations based on your human emotions. Citing internal documents and an unnamed source, Bloomberg reports that the company has designed a device that you wear like a wristwatch and beta testing is apparently underway. “Eventually the technology could be able to advise the wearer how to interact more effectively with others,” reads the report.
It’s unclear how far along Amazon is in developing this new frontier of Alexa-powered surveillance. Bloomberg could not say whether the testing in question “includes prototype hardware, the emotion-detecting software or both.” The report does say that the device is a “wrist-worn gadget” being developed in collaboration with Lab126, the same company that helped develop the Echo and the Fire phone (LOL). And that’s about all the information we’ve got on the actual hardware. However, Amazon does hold a number of patents that shed some light on how technology like this might work.
One such patent includes the image above. (We’ve added some colour to the original drawing to make it more fun and realistic.) This patent, titled “Voice-based determination of physical and emotional characteristics of users,” describes how an Alexa device could receive input from a user through microphones and tag it accordingly.
As the lifelike drawing above illustrates, a sick woman could arrive home, cough, and say she’s hungry. Alexa would tag her as “sick” and then recommend chicken soup. Alexa could even offer to order cough drops from Amazon in an effort to ease the human’s malaise. The Alexa workflow, according to the same drawing, might look a little bit like this:
This example seems innocuous enough, but Amazon seems to have bigger plans than cough drops for this emotion-reading technology. Another patent awarded on April 30 of this year describes more advanced systems for analysing not only the emotions of a user but also their mental health state.
This patent describes “a method for identifying human mental health states using mobile device data” that would come from “a mobile device (e.g. smartphone, wearable device).” The patent goes on to explain how measurements that are taken from the device—everything from accelerometer and GPS data to text message logs and voice data—could be used to determine “the likelihood of the one or more symptoms being present in the user.” Symptoms, in this instance, include those outlined in the DSM-V, such as those associated with depression or PTSD. Here’s part of a chart from that patent application:
This most recent patent goes on to describe how audio could also be used to determine the mental health state and emotions of a user. Here, the technology analyses characteristics like articulation, pitch variation, and vocal effort (“a measure of a feature designed to discriminate soft, breathy voices from loud, tense ones”) in order to provide more data. The application stops short of including drawings of humans feeling depressed or suffering from PTSD symptoms.
The voice-based aspect of this technology appears to be based, in part, on another Amazon patent that was made public last October. This one describes the process of using voice input to determine “happiness, joy, anger, sorrow, sadness, fear, disgust, boredom, stress, and other emotional states” in the user.
This patent also mentions how detecting “a certain voice feature, such as a cough or sniffle” could help the software tag the user as having “a specific physical or emotional abnormality.” All of these terms sound a bit cold and clinical, though if you’ve ever read a patent filing, you know that this is the vernacular of such documents.
It’s hard to know what Amazon really plans on doing with all of this technology, as hypothetical as some of it seems. However, it does look like Amazon might finally putting this patented technology to use in the development of some new hardware. We also know that Amazon is ramping up its interest in health-related businesses. Last June, Amazon put Nader Kabbani, a veteran of the Kindle team, in charge of the company’s burgeoning pharmaceutical business earlier this year. One can only imagine how useful detailed information about the symptoms of millions upon millions of Alexa users would be for this new business.
Sounds creepy, though, doesn’t it? Imagine if you plugged in a brand new Echo, sniffle, and then Alexa recommended you buy some Kleenex. That’s actually a fairly quaint application for the emotion-detecting technology that Amazon is working on. One could easily imagine a future in which Alexa devices could be used to monitor people’s mental health state and, potentially, make errors. We know from countless reports of Echo errors that Amazon’s artificially intelligent assistant isn’t always as smart as it’s supposed to be.
And we certainly know that Amazon isn’t as good at paying attention to user privacy as it should be. Maybe more than anything, it’s just deeply unsettling to think that technology could soon be surveilling our physical existence while also monitoring our emotions and mental health.
There’s no need for alarm at this point in time, however. Technology that better helps us understand ourselves, our emotions, and our mental health is an undoubtedly promising idea. It just sort of sucks that Amazon might be the first to market with a gadget and with software that very well might do this poorly in order to collect more data and sell more cough drops.
We’ve reached out to Amazon for more information on reports of its emotion-detecting wearable device and the software behind it. We’ll update this post if we hear back.