A UK startup says it has the answer to teens and kids sexting and sending nude photos: Continual censorship and surveillance.
Image: SafeToNet
SafeToNet is rolling out a new anti-sexting feature as part of its subscription plan, which will filter inappropriate messages from children’s phones, the Telegraph reports.
If the app identifies an incoming nude photo, it will censor it and then prohibit a kid from using their phone’s camera “for up to 20 minutes” in an effort to prevent them from responding with an explicit photo. It also reportedly locks the phone’s camera if a kid writes a text that the app deems “inappropriate”. If the kid keeps at it, the app can prevent them from using the camera for a longer period of time, and if they attempt to send an explicit message four times within a day, the company will inform their parents.
The anti-sexting feature isn’t available yet, but the company is demoing it this week at Mobile World Congress and, according to TechCrunch, it will roll it out in the UK in April.
The app works through the use of a custom keyboard, which uses machine learning to identify the “inappropriate content,” according to the company, and it scans a phone’s photo library to flag explicit images, which SafeToNet’s website says “will be edited to render it un-viewable”.
It’s unclear what exactly constitutes an inappropriate message. We have reached out to SafeToNet for more information on words and phrases that would be flagged, and if the app is able to identify slang and emoji. According to SafeToNet, its Safeguarding Keyboard uses AI and “a machine learning database” to identify sexting language, which does include emojis. But the words and phrases in an image on the website range from the supremely broad (“send it!” and “show me”) to the supremely obvious (“naked” and “nudes”).
It’s hard to imagine teens wouldn’t be able to figure out a way to circumvent such a feature, given the software is run on an AI system, and artificial intelligence is, to date, pretty terrible at picking up on the nuances of human language. While perhaps a teen under parental surveillance might not be able to text something along the lines of “take your shirt off” or “I want to see your dick” without being flagged, they may be able to text in code and fly under the radar. It also remains to be seen whether the machines identify “u up”.
The app is always running in the background of the phone, according to TechCrunch, which sounds both incredibly invasive and also like a major battery suck. That’s how it’s able to detect potentially sexty behaviour in realtime. CEO Richard Pursey also told TechCrunch that the app, using AI, can even learn a kid’s phone behaviour and inform parents if their kids might be showing signs of depression or aggression.
SafeToNet isn’t the first company to try to prevent kids from sexting. ZipIt, an app from Childline, gives kids a GIF guide on how to respond to unwanted solicitations for nude images. Detoured is an app that scans all of the images on a child’s phone and, if it flags any as explicit, will ensure the images are “quarantined”, or unable to be sent from the device elsewhere. SafeToNet seemingly wants to come across as a less dictatorial option for parents by noting that kids are aware of the software on their phone and have to grant the app permission to share any of their content with parents – unless they repeatedly ignore warnings.
As Pursey told TechCrunch:
Children are trying to determine their identity, they’re trying to work out who they are but … we’re not there to be the parent. We’re they’re to advise, to do the safeguarding. … But [parents’ job] is to try and make sure that their children are well-balanced and well-informed, and can handle the challenges that life brings.
Our job is certainly not to police them – quite the opposite. It’s to enable them, to give them the freedom to do these things. Rather than sledgehammer to crack a nut, which is the existing parental control systems. In my opinion they cause more harm than they actually save or protect. Because parents don’t know how to use them.
It’s somewhat laughable for the company to believe that it isn’t policing children while its software flags anything potentially inappropriate and has the ability to lock their cameras when deemed necessary. It’s also worth noting that machines designed to flag nudes have proven screwy, and that this might also prevent kids from looking up images and information related to inquiries about sexual education.
It’s almost as if constant surveillance and automated censorship isn’t the solution to preventing kids from engaging in unwanted sexual activities online. And with nearly unbridled online access these days, it’s a futile effort.