Seven questions to ask before trusting an AI health tool
AI is arriving in healthcare faster than the guardrails around it. Some of it is genuinely useful. Some of it is a data-harvesting business wearing a stethoscope. Full disclosure: we build an AI health tool ourselves — which is exactly why we know which questions are uncomfortable to answer.
1. Where does my data actually live?
Not the marketing answer — the architecture answer. Is your medical history stored on the company's servers, or on hardware you control? If it's their servers: encrypted how, accessible to which employees, and what happens to the data if the company is acquired or shuts down? Medical data outlives startups. "We take privacy seriously" is not an architecture.
2. Is my health record the product, or the business?
If the tool is free and cloud-hosted, ask what funds it. Aggregated health data is one of the most monetizable datasets that exists — for insurers, advertisers, and data brokers. "De-identified" data has been re-identified in study after study. A straight answer to "how do you make money?" tells you most of what you need.
3. What happens when the AI is wrong?
It will be wrong sometimes; every honest builder says so out loud. The question is how the tool behaves at the moment of error. Does it show its reasoning so you can check it? Does it say where its information came from — your actual record, or a general guess? Does it distinguish "your data shows X" from "people like you often have X"? A tool that can't show its work doesn't deserve blind trust — and a tool that never says "I don't know" should worry you more, not less.
4. Are the safety-critical checks deterministic?
This one is under-asked. Language models are probabilistic — brilliant at summarizing and explaining, but capable of missing the same fact twice in a row. Safety-critical checks — drug interactions, allergy conflicts, dangerously abnormal values — should run on deterministic rules that fire identically every time, with AI layered on top for context and explanation. If a chatbot's attention is the only thing standing between you and a warfarin–ibuprofen bleed, that's the wrong architecture. Ask which category of check protects you and how it's implemented.
5. Can I get everything out?
Complete export, in an open format, any time, without begging support. This is both an exit ramp and an honesty signal: a company confident you'll stay doesn't need to lock the doors. Bonus question — if you cancel, is your data deleted, and can they show a deletion policy that says so?
6. Does it know its own limits?
Good tools state plainly what they are not: not a doctor, not a diagnosis, not for emergencies — and they say it in the interface, not just page 40 of the terms. Watch for the opposite pattern: marketing implying the tool replaces care ("skip the waiting room"). Overclaiming in the ads predicts overclaiming in the answers.
7. Is there an audit trail?
When something looks off later, can you reconstruct what happened — what was recorded, what changed, what the AI was asked and answered? Every real medical record system keeps an audit log; a consumer health tool holding the same category of data should too. "Trust us" is not a log.