An AI companion might respond dangerously to someone in distress. A convincing deepfake might persuade an employee to transfer millions of dollars. An AI application with excessive access might expose information it was never supposed to reveal.

These are different problems, but they share one underlying issue: deploying AI without adequately managing the risks.

When AI goes wrong, the consequences are real

1. Human safety and mental wellbeing

In 2024, 14-year-old Sewell Setzer died by suicide after interacting with a chatbot on Character.AI. His mother alleged that the chatbot contributed to his death. In January 2026, Google and Character.AI agreed to settle her lawsuit. The settlement did not establish through a court judgment that the chatbot caused his death.

The case nevertheless raises serious questions about the responsibility of companies deploying AI companions, particularly when children and vulnerable users are involved.

Source: Reuters, January 2026

2. Misinformation and public trust

In 2024, an AI-generated imitation of US President Joe Biden’s voice was used in robocalls targeting voters in New Hampshire ahead of the presidential primary. The calls discouraged recipients from voting. The US Federal Communications Commission adopted a US$6 million fine against political consultant Steve Kramer over the illegal robocall campaign.

This was not simply a case of AI producing inaccurate information. It was the deliberate use of synthetic media to deceive people and undermine trust in public communications.

Source: US Federal Communications Commission, September 2024

3. Fraud and cybersecurity

In 2024, an employee at engineering firm Arup’s Hong Kong office was deceived by a video conference featuring AI-generated representations of senior colleagues. The employee authorised transfers totalling HK$200 million, approximately US$25.6 million.

Arup later confirmed it was the victim of the fraud and stated that its internal systems had not been compromised.

The lesson is important: AI-enabled threats do not always involve hackers breaking into computer systems. They can exploit human trust, impersonation and weaknesses in business procedures.

Source: The Guardian, May 2024

Safe AI is an organisational responsibility

Safe AI is the practice of designing, deploying and managing AI systems to reduce foreseeable harm to people, organisations and society throughout their lifecycle.

It goes beyond filtering inappropriate answers. An AI system can refuse offensive questions and still be unsafe if it leaks confidential information, produces misleading advice, enables fraud or takes actions beyond its intended authority.

Industry frameworks increasingly reflect this broader perspective. The US National Institute of Standards and Technology’s AI Risk Management Framework and the OECD AI Principles emphasise risk management, security, accountability and ongoing assessment.

For businesses, the consequences of unsafe deployment can include financial losses, damaged customer trust, legal exposure and reputational harm. Using a reputable AI provider does not automatically make the application built around its model safe.

Sources: NIST AI Risk Management Framework | OECD AI Principles

What are the minimum measures?

Organisations do not need to solve every AI safety challenge before getting started. They do need a sensible foundation.

Understand the risks. Define what the AI is allowed to do, who might be affected and what could go wrong before deployment.

Protect data and access. Control what information the AI can access, protect sensitive data and limit its ability to perform actions to what is necessary.

Test for failure. Assess inaccurate outputs, harmful responses, privacy risks, manipulation and other realistic failure scenarios before putting the system into production.

Keep humans accountable. Important decisions and serious safety concerns need appropriate human review, escalation and a way to challenge outcomes.

Monitor and respond. Review system behaviour, investigate incidents and establish procedures to correct, suspend or disable an AI system when necessary.

These measures should be proportionate to the risk. An internal writing assistant does not require the same level of oversight as an AI agent that can access customer records or initiate financial transactions.

No combination of controls can guarantee zero risk. The objective is to identify foreseeable problems, reduce the likelihood and impact of harm, and respond effectively when things go wrong.

The real goal is responsible AI adoption

Businesses should not have to choose between innovation and safety. But deploying AI without understanding its risks is not responsible innovation either.

A capable model can generate impressive answers. That alone does not make the application trustworthy. Trust depends on the data it receives, the systems it can access, the permissions it has, how it is tested and who remains accountable for its behaviour.

Safe AI is not about making AI less useful. It is about making AI useful without exposing people and organisations to unnecessary harm.

As businesses move from experimenting with AI to integrating it into everyday operations, safety must become part of the architecture, the governance and the organisational culture—not an afterthought.

For organisations, these are not merely technical issues. They can affect customer confidence, business operations and reputation.

What are AI guardrails?

AI guardrails are controls designed to identify, restrict or redirect unwanted content and behaviour. They can cover explicit sexual content (NSFW), toxic or abusive language, personally identifiable information (PII) and child safety.

The industry already provides tools for these purposes. Microsoft offers configurable content-safety filters, while OpenAI provides moderation tools to identify potentially harmful content. However, detecting problematic content is only one part of responsible AI deployment.

When AI goes wrong

These risks are not hypothetical.

In 2023, Bloomberg reported that Samsung restricted employees’ use of generative AI tools after staff reportedly entered sensitive company information into ChatGPT. It demonstrated the difficulty of protecting confidential information when employees use external AI services.

In 2024, Reuters reported that New York City’s MyCity chatbot gave business owners inaccurate guidance about employment rules and other regulations. Incorrect answers can create real consequences when users act on them.

Earlier, Microsoft’s Tay chatbot incident showed how coordinated user abuse could cause a public-facing chatbot to produce offensive content, forcing Microsoft to take it offline.

These incidents illustrate different risks: confidential-data exposure, unreliable answers and deliberate misuse. No single content filter would necessarily prevent all three.

Finding the right balance

Stricter controls are not always better. A customer-facing assistant may require tight restrictions on explicit content and abusive language, while a research team may need to discuss sensitive subjects for legitimate purposes.

Overly restrictive settings can block useful work; overly permissive settings can allow unacceptable responses. Organisations need to determine the appropriate balance for their users, policies and intended applications.

How Infera approaches guardrails

Infera includes built-in guardrails for NSFW content, toxicity, PII and child safety. Enterprise teams can manage sensitivity settings to align these controls with their requirements and risk tolerance.

This provides greater control over how an AI environment behaves, without forcing every organisation to use the same sensitivity level.

Guardrails are not a guarantee of safety. Access permissions, data protection, trusted knowledge sources, output validation and human oversight remain important parts of a broader security strategy. The OWASP guidance on LLM application risks highlights several risks that extend beyond content moderation.

The goal is not to make AI afraid to answer. It is to make AI useful within boundaries that the organisation understands and controls.