Anthropic has raised eyebrows by announcing it will ban users from “sustained and needless” abusive behaviour towards its AI.
The Claude-maker said on Thursday it was updating its usage policy to let its tools end interactions where people are “cruel” – something it said it had done in “rare” cases prior.
It will now be included on a list of forbidden conduct which also contains bullying others, promoting self-harm and creating non-consensual intimate imagery.
While Anthropic said the policy would only apply in “extreme cases” of repeated abuse, it has reignited debate over how we should communicate with AI tools.
The company said it would not apply to “common versions of user frustration, pushback, dark creative themes, or model testing and research” – suggesting it would only apply in clearly deliberate instances.
But what Anthropic would consider “abusive or cruel behaviour” towards its models remains unclear.
Screenshots of the new policy have been widely shared and commented on across social media – with some praising it as advancing good manners or “model welfare”.
But others criticised the plan, with one person calling it “deeply wrong, external” and another suggesting it might undermine abuse directed at humans and animals.
“You can’t be cruel to numbers and maths,” wrote Dr Barry Scannell, technology partner at Irish law firm William Fry, is LinkedIn, external.
“This level of anthropomorphisation of AI is harmful. It leads people to believe that it’s something it’s not.”
Microsoft AI director Mustafa Suleyman, who in September criticised rival firm Anthropic for treating AI like it is human, reacted to Anthropic’s update with a melting face emoji.