Ask a popular AI chatbot to cite the section number of a specific Australian consumer law and there’s a decent chance it will give you one, cleanly formatted, entirely wrong, and delivered with the same flat confidence it uses for the correct ones. That’s not a bug report on one app. It’s just how these things work, and it’s the single most important thing to understand before you let one near anything that matters, from a tax question to a medical symptom to a dispute with a retailer.
We get asked about this constantly, usually by someone who got burned once and wants to know if they can trust the tool again. Short version: yes, carefully. The longer version is worth your time, because the habit of checking is the whole game.
What a hallucination actually is
In AI circles the industry term is “hallucination”, which is a slightly odd word for what’s happening but it’s the one that stuck. It means the model generates something false, or something that sounds plausible but has no basis in fact, and presents it as if it were true.
The thing to get your head around is that a chatbot isn’t a search engine with better manners. It isn’t looking something up and reporting back. It’s predicting, word by word, what a good answer would probably look like, based on patterns learned from enormous volumes of text. Most of the time that prediction lines up with reality, because reality is what most of that text described. But the system has no internal concept of “I don’t actually know this.” It just keeps predicting the next plausible word, and a fabricated court case or a non-existent statistic comes out formatted exactly like a true one, because statistically, that’s what true ones look like too.
Our take: the word “hallucination” does the industry a favour it doesn’t deserve, because it makes the behaviour sound like an occasional glitch rather than the system working as designed. It’s closer to confident guessing dressed up as recall.
Why it happens more than people expect
A few things push these errors up. Obscure or niche topics are riskier, because the model has seen less reliable training material to draw on. Anything requiring precise numbers, names, dates or citations is riskier again, since the model is generating a plausible-shaped answer rather than retrieving a stored fact. And questions about events after the model’s training cut-off are a near-guarantee of trouble unless the tool is explicitly set up to search the live web and tell you so.
There’s a chip and hardware angle here too, for readers who follow that side of things. The current generation of on-device AI features, the kind running locally on phones built around chips compared in our Snapdragon vs MediaTek vs Apple A-Series rundown, are generally doing smaller, narrower tasks like photo search or text summarising. The big general-purpose hallucination risk mostly comes from the large cloud-based models, not the small on-device ones, because the small ones are deliberately kept to tasks where guessing is less consequential.
We’d also argue, and this is the mildly unfashionable bit, that a lot of the “AI is getting smarter every month” coverage glosses over the fact that hallucination hasn’t been solved and arguably can’t be fully solved with the current approach. Newer models hallucinate less often on average, sure. But “less often” on a tool millions of people now use for homework, health questions and small business admin still adds up to a lot of wrong answers landing in front of people who have no way to tell.
Where this bites in an Australian context
We’ve seen reports of chatbots inventing ACCC guidance that doesn’t exist, misquoting NBN plan terms, or confidently stating superannuation and tax rules that were true in a different country or a different year. None of these come with a warning label. The chatbot doesn’t say “this bit I’m less sure about.” It says it all in the same tone.
This matters more here than it might elsewhere because Australians are increasingly asking these tools about genuinely consequential things: whether a contract clause is enforceable, whether a text message is a scam, what Centrelink or ATO thresholds currently apply. Our State of Consumer AI in 2026 piece found this kind of everyday, practical use is exactly where adoption has grown fastest among ordinary Australians, not in novelty chat. That’s also exactly the use case where a confidently wrong answer does the most damage.
If you’re trying to work out whether something you were told, by a chatbot or by a text message, is legitimate, the same scepticism habits that work for scam detection apply. Our guide on how to spot a scam website covers a lot of the same “does this actually check out” instincts you want to bring to an AI answer.
The tell-tale signs worth watching for
A few patterns crop up often enough to be useful warning signs, not proof, but worth a second look.
- Oddly specific numbers, dates or names with no source attached, especially legal citations, academic references or statistics quoted to a precise decimal.
- Total confidence on a question that’s genuinely contested or recent, where a knowledgeable human would hedge or say “it depends.”
- An answer that changes meaningfully if you ask the same question again in a new chat, which suggests the model is pattern-matching rather than retrieving something fixed.
None of these guarantee an error. Plenty of correct answers are also specific and confident. But when two or three line up at once, that’s your cue to go and check independently rather than act on it.
How to actually catch it before it costs you
The practical fix isn’t complicated, it’s just a habit most people haven’t built yet, the way we didn’t build the habit of checking a URL before clicking it until phishing taught us the hard way.
Treat anything with a name, number, date or citation attached as a claim to verify, not a fact to repeat. If a chatbot tells you a regulation, a product spec or a historical date, search for the primary source yourself before you rely on it. For anything consumer-related in Australia, that usually means going straight to the regulator rather than trusting a paraphrase: the ACCC’s own guidance pages for consumer law questions, or the Australian Cyber Security Centre for anything security-related.
Ask the chatbot to show its working, or its sources, where the tool supports it. Some newer AI assistants and AI-enabled browsers, the kind we tested for our AI web browsers compared piece, will pull from live web pages and link to them, which at least gives you something to check. That’s meaningfully safer than a model answering purely from memory, though it’s not foolproof either, since the model can still misread or misquote the page it’s citing.
And for anything genuinely high-stakes, money, legal standing, a medical decision, treat the chatbot as a drafting assistant or a starting point for research, not as the final word. That’s a boring piece of advice. It’s also the one that actually prevents the expensive mistakes.
A quick gut check before you act on an AI answer
Ask yourself three things before you rely on what a chatbot just told you: would this answer embarrass me if it turned out to be wrong, is there a primary source I could check in under two minutes, and did the model hedge at all or state it as flat fact? If the stakes are low and the answer is plausible, fine, proceed. If any of those three raise a flag, go check.
What the companies building these tools are actually doing about it
To be fair to the industry, this isn’t being ignored. Newer model releases from the major AI labs have generally reduced hallucination rates on standard benchmark tests compared with their predecessors, and features like live web search, citation links and “I’m not certain about this” hedging language are becoming more common in mainstream tools. Some assistants now explicitly flag lower-confidence answers rather than stating everything in the same tone.
But benchmark improvements are measured on curated test questions, not on the messy, specific, Australian-context questions real people actually ask, like whether a particular telco’s fine print allows an early exit fee, or what the current threshold is for a specific government rebate. There’s still a gap between “the industry says hallucination rates are trending down” and “you can stop checking.” We’re not there yet, and we’d argue anyone telling you otherwise is spruiking rather than reporting.
Our verdict on trusting chatbot answers
These tools are genuinely useful for drafting, brainstorming, summarising something you can personally verify, and explaining concepts in plain English. They’re a liability the moment you treat a specific factual claim as settled just because it was delivered smoothly. The chatbot doesn’t know the difference between confident and correct, and until that changes, you’re the quality control. Keep that habit and the tools are worth having. Drop it, and you’re one fabricated citation away from a bad decision made with total conviction.
— Dale Whitfield, Founding Editor, Tech Geek
