The biggest risk with AI isn’t that it gives a bad answer. It’s what happens when people trust that answer and act on it.
An AI companion might respond dangerously to someone in distress. A convincing deepfake might persuade an employee to transfer millions of dollars. An AI application with excessive access might expose information it was never supposed to reveal.
These are different problems, but they share one underlying issue: deploying AI without adequately managing the risks.
When AI goes wrong, the consequences are real
1. Human safety and mental wellbeing
In 2024, 14-year-old Sewell Setzer died by suicide after interacting with a chatbot on Character.AI. His mother alleged that the chatbot contributed to his death. In January 2026, Google and Character.AI agreed to settle her lawsuit. The settlement did not establish through a court judgment that the chatbot caused his death.
The case nevertheless raises serious questions about the responsibility of companies deploying AI companions, particularly when children and vulnerable users are involved.
2. Misinformation and public trust
In 2024, an AI-generated imitation of US President Joe Biden’s voice was used in robocalls targeting voters in New Hampshire ahead of the presidential primary. The calls discouraged recipients from voting. The US Federal Communications Commission adopted a US$6 million fine against political consultant Steve Kramer over the illegal robocall campaign.
This was not simply a case of AI producing inaccurate information. It was the deliberate use of synthetic media to deceive people and undermine trust in public communications.
Source: US Federal Communications Commission, September 2024
3. Fraud and cybersecurity
In 2024, an employee at engineering firm Arup’s Hong Kong office was deceived by a video conference featuring AI-generated representations of senior colleagues. The employee authorised transfers totalling HK$200 million, approximately US$25.6 million.
Arup later confirmed it was the victim of the fraud and stated that its internal systems had not been compromised.
The lesson is important: AI-enabled threats do not always involve hackers breaking into computer systems. They can exploit human trust, impersonation and weaknesses in business procedures.
Source: The Guardian, May 2024
Safe AI is an organisational responsibility
Safe AI is the practice of designing, deploying and managing AI systems to reduce foreseeable harm to people, organisations and society throughout their lifecycle.
It goes beyond filtering inappropriate answers. An AI system can refuse offensive questions and still be unsafe if it leaks confidential information, produces misleading advice, enables fraud or takes actions beyond its intended authority.
Industry frameworks increasingly reflect this broader perspective. The US National Institute of Standards and Technology’s AI Risk Management Framework and the OECD AI Principles emphasise risk management, security, accountability and ongoing assessment.
For businesses, the consequences of unsafe deployment can include financial losses, damaged customer trust, legal exposure and reputational harm. Using a reputable AI provider does not automatically make the application built around its model safe.
Sources: NIST AI Risk Management Framework | OECD AI Principles
What are the minimum measures?
Organisations do not need to solve every AI safety challenge before getting started. They do need a sensible foundation.
Understand the risks. Define what the AI is allowed to do, who might be affected and what could go wrong before deployment.
Protect data and access. Control what information the AI can access, protect sensitive data and limit its ability to perform actions to what is necessary.
Test for failure. Assess inaccurate outputs, harmful responses, privacy risks, manipulation and other realistic failure scenarios before putting the system into production.
Keep humans accountable. Important decisions and serious safety concerns need appropriate human review, escalation and a way to challenge outcomes.
Monitor and respond. Review system behaviour, investigate incidents and establish procedures to correct, suspend or disable an AI system when necessary.
These measures should be proportionate to the risk. An internal writing assistant does not require the same level of oversight as an AI agent that can access customer records or initiate financial transactions.
No combination of controls can guarantee zero risk. The objective is to identify foreseeable problems, reduce the likelihood and impact of harm, and respond effectively when things go wrong.
The real goal is responsible AI adoption
Businesses should not have to choose between innovation and safety. But deploying AI without understanding its risks is not responsible innovation either.
A capable model can generate impressive answers. That alone does not make the application trustworthy. Trust depends on the data it receives, the systems it can access, the permissions it has, how it is tested and who remains accountable for its behaviour.
Safe AI is not about making AI less useful. It is about making AI useful without exposing people and organisations to unnecessary harm.
As businesses move from experimenting with AI to integrating it into everyday operations, safety must become part of the architecture, the governance and the organisational culture—not an afterthought.