Most of the on-device AI conversation in 2026 has been framed as a privacy upgrade. Apple Intelligence runs locally, your prompts don’t leave the device, the cloud is no longer the default — what’s not to like? It’s a reasonable starting point, and as our editor Dale would be the first to say, the broad shift from “send everything to a US data centre” to “process it on the chip in your pocket” is genuinely good news. But it’s not the end of the privacy conversation. It’s the start of a different one. After spending the last few months looking carefully at where on-device AI actually puts your data, the team’s honest view is that the marketing has overshot the reality. On-device AI is a meaningful privacy win in some places, a privacy wash in others, and an outright new privacy risk in a couple of specific cases that nobody’s talking about loudly enough.
What follows is the plain-English version of what we’d want our parents and our small-business clients to actually understand before they assume the new locally-run AI features on their phone or laptop are as private as the keynote made them sound.
What “on-device” really guarantees, and what it doesn’t
The pitch for on-device AI is simple: if the model runs on your phone, your prompt never leaves the device, so there’s no server-side log, no third-party training risk, no overseas data transfer. All of that is true at a basic mechanical level, and for the discrete on-device inference call — the moment of “give me a summary of this email” — it is genuinely a privacy improvement over cloud AI.
The problem is that the inference call is only one small part of the full picture. To make the on-device AI feel useful, the model needs context — what’s on your screen, what’s in your photos, what’s in your messages, what’s on your calendar, who’s in your contacts. The way Apple, Google and Microsoft have solved this is to give the on-device AI broad, system-level read access to that personal data, indexed in real time by background services. That index lives on the device, yes. But it now exists in a structured, queryable form that didn’t exist before.
Put another way: pre-AI, your photos were a collection of files. Post-AI, your photos are a structured database of who appears where, when, doing what — sitting on the device, ready to be queried. That’s not a problem in itself. It becomes a problem when the device gets lost, when an attacker gets local access, when a backup gets compromised, or when a future feature decides to send aggregated summaries somewhere for “quality improvement”. The attack surface is fundamentally larger than it was on a pre-AI phone, even if no single byte ever leaves it.
The five real privacy risks worth paying attention to
Boiling down a fair bit of reading and testing, here are the genuine concerns Australians should actually think about, in rough order of how worried Dale is about each one:
1. The index is more valuable than the files. When on-device AI indexes your screen, photos, messages and documents into a vector database it can query in natural language, that index becomes one of the most concentrated stores of personal information ever to exist on a consumer device. A traditional malware author would have had to extract and parse a hundred different file types to build the kind of personal profile that’s now sitting pre-baked in a folder somewhere on your phone. Apple, Google and Microsoft all encrypt these indices at rest. That helps. It does not eliminate the issue.
2. Cloud fallback is the part nobody reads. Every major on-device AI implementation has a cloud fallback for queries the local model can’t handle. Apple’s Private Cloud Compute is the cleanest example — with cryptographic attestation about what the server can retain — but Samsung, Google and Microsoft have less formally-bounded versions of the same thing. In every case, the user gets a small notification or a setting toggle, and in every case the average user has no real sense of which of their queries went where. The on-device privacy guarantee depends on the user understanding when it applies and when it doesn’t. Most don’t.
3. Aggregated telemetry isn’t nothing. “We don’t send your prompts to our servers” doesn’t mean “we send nothing about your AI use to our servers”. Usage telemetry — which features you used, how often, how long, with what kind of input length — is still collected by default and still gets fed back to product teams. None of that is identifying in the obvious sense, but it’s a fingerprint of how you use your device. If you’re a journalist, a lawyer, a healthcare worker or anyone whose patterns themselves are sensitive, the telemetry is worth turning off where you can.
4. Family accounts, work accounts, shared devices. The on-device AI builds its index from whatever’s on the device. On a shared iPad, that means the parent’s email and the kid’s homework get mixed into the same index. On a BYOD laptop, that means your personal photos and your client’s contract documents get summarised by the same model with the same context. Apple’s per-account separation helps; Google and Microsoft are catching up. But the default configurations in 2026 still treat the device as a single context, and that’s a meaningful privacy issue for anyone whose device isn’t single-user.
5. Lost or stolen devices are now richer targets. A pre-AI smartphone, lost in a taxi, is a problem because the screen lock might be brute-forceable. A post-AI smartphone is a problem because the indexed personal database makes everything on it more accessible to anyone who gets past the lock screen. The countermeasure isn’t different — strong device passcode, biometrics, remote wipe enabled — but the urgency is higher than it was. Our team’s piece on spotting whether your phone has been compromised is, frankly, more important reading in 2026 than it was even a year ago.
What Australian law actually requires
The legal frame matters because it’s the bit that small business owners in particular keep missing. The Privacy Act applies to personal information regardless of where the processing happens — local or cloud. The Office of the Australian Information Commissioner’s guidance on AI has been increasingly direct: APP entities need to understand what personal information their devices are indexing, what’s being shared with third parties (including AI vendors), and what controls are in place. The fact that an AI model runs on-device doesn’t take an organisation outside the Privacy Act’s reach.
For consumers — and particularly for parents — the eSafety Commissioner’s resources on AI safety are worth knowing about. The Commissioner has been particularly focused on the way AI features process kids’ conversations, photos and chat logs, and the risks compound when the device is shared. The on-device-versus-cloud distinction is important here too, but the underlying point is the same: just because processing is local doesn’t mean it’s safe or appropriate. Those are different questions.
Practical steps that actually move the needle
The good news is that the major platforms have, mostly under regulatory pressure, given users real controls. The bad news is that the defaults are not configured for privacy. The steps that genuinely matter, in rough priority order:
- Audit which AI features are actually on. On iOS, Android and Windows, the Settings app now has an AI section. Walk through it. Turn off anything you don’t actively use. The “AI summarises my notifications” feature in particular should be a conscious choice, not a default.
- Set on-device-only mode where it exists. Most major Android phones now have a toggle to disable cloud fallback for AI features. Samsung calls it “Process AI features on-device only”. Pixel has a similar option. Use it.
- Don’t share devices between users without separating accounts. Family sharing on iOS, multiple users on Android, separate Windows accounts — these aren’t optional anymore if you care about which AI sees which data.
- Strong device locks are doing more work than they used to. Long passcodes, biometrics, remote-wipe enabled. This was hygiene before. It’s load-bearing now.
- Telemetry off where the regulator allows it. Apple’s “Share With Apple” toggles, Google’s “Send usage data” toggles, Microsoft’s diagnostic data settings. Set to minimum.
- Backups encrypted and held privately. An encrypted iCloud or Google backup is fine. An unencrypted local backup on a NAS with no auth is a much bigger problem in 2026 than it was a few years ago — that’s a complete copy of the AI-indexed database sitting somewhere you might have forgotten about.
- Treat AI features in shared work contexts as out-of-scope by default. If you’re a small business handling client data, the on-device AI on a BYOD laptop is not the same risk profile as a managed corporate laptop with policy-controlled features. The ACCC’s guidance on small business obligations overlaps with the OAIC’s here, particularly around vendor due diligence.
The honest framing: better, but not “solved”
None of this is a “don’t use on-device AI” argument. The features are useful, the privacy story is genuinely better than the cloud-only alternative, and the platforms have done meaningful engineering work to make local processing the default in places it used to be cloud. We’re not contrarians for the sake of being contrarians. But the way the technology has been marketed — and the way most users will read it — suggests a finished privacy story, and that’s not what’s been delivered.
What’s actually been delivered is a meaningful upgrade in one dimension (cloud transmission risk) traded against a smaller-but-real downgrade in another (the value of the device itself as a target). For most people, in most contexts, that trade is a net win. For a journalist with sensitive sources, a healthcare worker on a BYOD device, a small business handling client matters, or anyone whose threat model is more specific than “general consumer privacy”, it’s worth thinking carefully about what to leave on, what to switch off, and what to keep off the device entirely.
If you want the broader picture on how the smartphone in your pocket has shifted into being a much more capable AI device than it was a year ago, our recent piece on getting more out of your phone’s day-to-day performance covers the hardware side, and the longer-arc look at why phones remain the most genuinely useful consumer tech we own covers the why. The privacy lens here is the missing third piece — and the one that gets the least attention in launch reviews.
Final thoughts
The privacy story for on-device AI in 2026 is genuinely better than the cloud-AI story it’s replacing — and genuinely worse than the marketing makes it sound. The cloud transmission risk has dropped meaningfully. The on-device attack surface has grown. Aggregated telemetry, cloud fallback, shared devices and lost-or-stolen scenarios all need fresh thinking that the platform messaging hasn’t really equipped users to do. Dale’s view, which the rest of the team shares, is that on-device AI is a clear win as long as users actually engage with the new settings, understand the cloud fallback, and treat the indexed personal database as the high-value target it now is. Don’t be the person who turns on every AI feature in 2026 because the marketing said it was private. Be the person who turns on the ones that match your life, off the ones that don’t, and gets the locks and the backups right. The new on-device features are good. The new on-device responsibilities are real. Both are worth taking seriously.
