Gen AI Innovation
Employees are probably going to be disciplined for not verifying AI output. And a government department has had to withdraw an official national policy document because of it.
That's not a hypothetical risk scenario I'm using to make a point. It happened.
What Actually Happened
In April 2026, South Africa's Minister of Communications and Digital Technologies withdrew the Draft National Artificial Intelligence Policy after it emerged that the document's reference list contained fictitious sources — citations that didn't hold up, most likely generated by AI and never checked. The Minister was direct about the cause: "the most plausible explanation is that AI-generated citations were included without proper verification." He also confirmed there would be "consequence management for those responsible."
Sit with that for a moment. A policy meant to govern the responsible use of AI in the country was itself undone by AI used irresponsibly. Not because the technology failed — the technology did exactly what generative AI does when nobody checks its work. It produced confident, plausible-sounding references. Some of them simply didn't exist.
This Isn't a Rare Mistake
It's tempting to treat this as an isolated embarrassment. It isn't. According to the South African Generative AI Roadmap 2025 — a research study by World Wide Worx — 67% of large enterprises are already using generative AI tools in some form. Only 13% of enterprises have implemented comprehensive guardrails covering safety protocols, privacy protection, and bias mitigation.
That gap — two-thirds already using the tools, barely one in eight actually guarding against the risks — is exactly the gap this policy withdrawal fell into. It's the same gap sitting inside most businesses right now, whether or not it's made headlines yet.
The Risk Underneath All of This: Hallucinations
I wrote about this risk before it made the news, in a post on the risks everyone using AI needs to understand and mitigate. Hallucination was one of the categories I flagged:
The risk: the AI confidently states a "fact" that is completely fabricated.
Example: asking for legal precedents and having the AI invent a court case that never happened.
Mitigation: ask it directly to check its own output for factual accuracy, to say when it's unsure rather than guess, and to provide source links where possible.
That's a simple instruction. It would very plausibly have caught what went wrong in the policy document, if anyone had thought to ask for it — and if anyone had followed up by actually checking the sources that came back.
The Part That Doesn't Change
Beyond any specific mitigation prompt, the underlying discipline is the same one I keep coming back to: you are always the expert, and AI output — especially anything high-stakes, anything that will sit in front of a client, a court, or the public — needs to be fact-checked by a human before it goes anywhere. Not sometimes. Every time.
The Good News
None of this means stepping back from AI. It means training people properly before they rely on it for anything that matters. With the right training and the right habits built in from day one — not bolted on after something goes wrong — these risks are entirely avoidable, and the genuine productivity gains AI offers are still there for the taking.
The organisations that get this right won't be the ones that ban AI out of caution, or the ones that hand it the keys without oversight. They'll be the ones that teach their people exactly where the risk sits, and build the habit of checking before anything goes out the door.