AI detectors claim to tell whether a text was written by a person or by a tool like ChatGPT. Schools, employers, publishers and search-minded site owners use them, but their results are far less reliable than the confident percentage scores suggest. This guide explains how detectors work, what the evidence says about their accuracy, and what to do instead of relying on a score.
Key takeaways
- Detectors estimate how “predictable” a text looks; they cannot see how it was written.
- Independent tests show both false positives (human text flagged) and false negatives (AI text missed).
- Non-native English writers and formulaic writing are flagged more often.
- Never treat a detector score as proof; combine it with drafts, version history and a conversation.
How AI detectors work
- Statistical signals: language models generate likely words, so AI text tends to be statistically predictable. Detectors measure things such as perplexity (how surprising the words are) and burstiness (variation in sentence length and structure).
- Trained classifiers: a model is trained on examples of human and AI text and learns to tell them apart.
- Watermarks: some AI providers embed hidden statistical patterns in the text they generate. This only works for text from models that use the watermark, and edits can weaken it.
How reliable are they?
The evidence is mixed at best:
- OpenAI released an AI text classifier in January 2023 and withdrew it in July 2023 because of its low accuracy. In its own evaluation it correctly identified about 26% of AI-written text and wrongly flagged about 9% of human-written text.
- A 2023 Stanford study published in Patterns tested seven detectors on essays by non-native English speakers and found they wrongly flagged more than 61% on average, while essays by US students were almost never flagged.
- Turnitin reports about 98% accuracy with under 1% false positives, but these are the vendor’s own figures and apply to documents with more than 20% AI-generated text.
- Simple paraphrasing or rewriting can lower detection scores, while formulaic human writing can raise them.
A detector result is a probability estimate from a model, not evidence of how a text was produced.
Who is most at risk of a false flag?
- Non-native English writers, whose vocabulary and sentence patterns can look statistically “simple”.
- Writers following strict templates, such as reports or legal text.
- People who use grammar tools or heavy editing.
- Short texts, where there is little signal to measure.
What to do instead
| If you are… | Better approach |
|---|---|
| A teacher or manager | Look at drafts and version history, ask the author to explain their work, and use assignments that include personal context or an oral follow-up |
| A publisher or site owner | Judge content on accuracy, originality and usefulness; check facts and sources rather than relying on a score |
| A writer worried about false flags | Keep drafts, notes and version history (Google Docs or Word history) as evidence of your process |
| Using AI to help write | Follow your organisation’s policy, disclose use where required and edit substantially; see dos and don’ts of generative AI |
How to spot AI-generated text yourself
- Generic phrasing and filler openings that say little.
- Repetitive structure, with every paragraph following the same pattern.
- Confident statements without sources, or references that cannot be found.
- Factual errors or invented quotes and links.
- Lack of personal experience, specific examples or a distinct voice.
None of these is proof, but they are better prompts for questions than a percentage. For how the models behind this text work, see LLMs vs generative AI and the main branches of AI.
Sources: CASRAI guide to AI detection accuracy in higher education; Times Higher Education on OpenAI shelving its detector.
Frequently asked questions
Can Google detect AI content?
Google has said it focuses on whether content is helpful, original and reliable rather than on how it was produced. Low-quality, mass-produced content performs poorly whether or not a person wrote it.
Are AI detectors accurate?
Not reliably. Independent studies show both false positives and false negatives, and results vary by tool, text length and writer. Treat scores as a weak signal.
Can AI detectors be fooled?
Yes. Paraphrasing, editing or prompting a model to write differently can reduce detection, which is one reason detectors cannot serve as proof.
Leave a Reply