How AI Detectors Work: Perplexity, Burstiness and Classifiers
What AI text detectors actually measure, the four main approaches, why they fail on real writing, and what an 'AI score' really means.
An AI detector cannot see who typed a document. It has no access to ChatGPT’s logs and no database of everything a model has ever produced. All it has is the text. So every detector on the market is doing the same fundamental thing: measuring how statistically similar a piece of writing is to typical language-model output, and reporting that similarity as a percentage.
Once you understand that, the rest follows: why detectors flag human writing, why they miss edited AI writing, and why no score should be treated as proof.
The four approaches
1. Perplexity
Perplexity measures how surprised a language model is by each word in a text. Run the text through a model, and at each position ask: how likely was the word that actually appeared? A text where every word is highly predictable has low perplexity. A text full of unexpected word choices has high perplexity.
Language models generate text by choosing likely words. So, by construction, their output has low perplexity when scored by a similar model. Human writing is messier: odd word choices, unusual constructions, in-jokes, typos, sentences that take a turn. That messiness reads as high perplexity.
The problem is obvious once stated. Plenty of human writing is highly predictable: legal boilerplate, lab reports, technical documentation, anything written in a second language, anything that has been through a heavy grammar checker. All of it scores low. Meanwhile, a model prompted to “write like a sarcastic teenager” produces high-perplexity text and sails through.
2. Burstiness
Burstiness is the variation in perplexity across a text, and in practice also covers variation in sentence length and structure. Human writers are bursty: a long complex sentence, then a short one. A paragraph of dense argument, then a one-liner. Models, especially with default settings, produce sentences of remarkably even length and complexity.
Burstiness is a better signal than raw perplexity because it is harder to fake by accident. But it fails on the same populations: formal and non-native writers tend toward uniform sentences, and it fails completely on short texts, where there is not enough variation to measure.
The 7 edits guide is, in effect, a guide to raising burstiness. That is not a coincidence: what detectors measure is also what makes text feel mechanical to readers.
3. Trained classifiers
The commercial detectors (Turnitin, GPTZero, Originality, Copyleaks and others) mostly use classifiers: neural networks trained on large sets of labelled examples, “this is human”, “this is GPT-4”, “this is Claude”, and asked to sort new text. A classifier can pick up on much subtler signals than perplexity alone: vocabulary frequencies, punctuation habits, paragraph structure, the presence of hedging phrases, how transitions are used.
Classifiers are the most accurate approach on text that resembles their training data. They are also the most opaque: they cannot tell you why a passage was flagged, only that it resembles the AI class. And they degrade whenever the generators change, so vendors retrain regularly, and scores on the same text drift over months. The number you get today is a function of the model version the vendor is running today, trained on the generators they had samples from.
4. Watermarking
Watermarking is different in kind. Instead of analysing text after the fact, the generator embeds a statistical signal at generation time, for example by subtly preferring words from a secret “green list” at each step. A detector with the key can then check whether a text has too many green-list words to be chance.
Watermarking is the only approach that can, in principle, give a confident answer. It has been demonstrated in research, and Google has described deploying a version (SynthID) for some of its own model outputs. But it is not widely deployed across the models people actually use, it only works if the detector has the key for the specific generator, and it can be weakened by paraphrasing. For the text you are likely to be checking, watermarking is not what the detector is using.
Comparison of approaches
| Approach | What it measures | Strengths | Fails on |
|---|---|---|---|
| Perplexity | Word-level predictability | Simple, model-agnostic, explainable | Formal, technical, non-native and edited human writing; AI text with unusual prompts |
| Burstiness | Variation in sentence length and predictability | Harder to trigger accidentally | Short texts; uniform human styles |
| Classifier | Learned patterns across many features | Highest accuracy on familiar generators | New models, edited AI text, out-of-distribution human writing; opaque |
| Watermarking | Hidden signal inserted at generation | Can be near-certain when present | Anything from a generator without a watermark, or when the key is unavailable; paraphrase |
Most commercial products combine the first three. None publishes a method in enough detail to verify their accuracy claims independently, and the independent evaluations that exist find substantial variation between tools and between text types. Treat any advertised accuracy figure as a marketing number measured under conditions you cannot check.
Why detectors fail, in both directions
False positives (human text flagged as AI) happen because the “human” class in training data does not cover all humans. The writers most affected are the ones whose natural style is closest to model output: non-native English writers, students taught to write in rigid templates, technical and legal writers, and anyone who runs their prose through aggressive editing tools. Several studies have documented elevated false-positive rates for non-native writers specifically, and the mechanism (lower perplexity, lower burstiness) applies to every detector built on these signals. See Does Turnitin Detect ChatGPT? for what this means in practice for students.
False negatives (AI text passing as human) happen because the generator can be steered. A model asked for a specific voice, given examples of the author’s writing, or set to higher randomness produces text outside the detector’s expected distribution. Editing does the same: change the sentence rhythm, swap the signature vocabulary, add specifics, and the statistical profile moves toward the human class. This is not a trick; it is what any good edit does.
Inconsistency happens because vendors have different models, training sets and thresholds. The same paragraph routinely scores 90% on one tool and 10% on another. Neither is “right”.
What an “AI score” actually means
When a detector says “82% AI”, it does not mean 82% of the words were written by a machine, and it does not mean there is an 82% chance the author used AI. Depending on the tool, it means one of:
- the classifier’s confidence that the document as a whole belongs to the AI class, or
- the proportion of sentences or passages that individually crossed the tool’s threshold, or
- a scaled version of a perplexity measure with no direct probabilistic meaning at all.
Vendors rarely say which. What the number is not is evidence about a person. It is a statement about the text’s resemblance to a pattern. A high score on a document with a full revision history, notes and drafts is a false positive. A low score on a document pasted in one go from a chat window is a false negative. The surrounding evidence outweighs the number in both cases.
What this means for you
The practical takeaways:
- Keep drafts. Revision history is the only evidence that reliably beats a detector score.
- Know your own profile. If your writing is formal, uniform and predictable, expect occasional flags whether or not you use AI. Vary sentence length; add specifics; use the words you actually say.
- Do not chase a score. Detectors are retrained. A trick that drops a number today may raise it next quarter. Text that reads naturally to a human is stable; text optimised for a detector is not.
- Use the signals as editing feedback. The AI Writing Pattern Checker shows you the vocabulary and structural patterns that these tools key on. That is useful even for fully human drafts, because those patterns are what make writing feel flat.
If you have an AI-assisted draft and are allowed to use it, the humanizer rewrites it toward natural rhythm and vocabulary. The goal is readability, not a number.
FAQ
Are AI detectors accurate?
They are useful and unreliable at the same time. On unedited output from well-known models they catch a lot. On edited AI text, on text from newer models, and on human writing that happens to be formal or non-native, they make frequent errors. No independent evaluation supports treating any detector’s output as proof, and the major vendors say so in their own guidance.
Can AI detectors detect paraphrased or humanized text?
Sometimes. Light paraphrasing leaves sentence structure intact, which classifiers often catch. Substantial rewriting with new specifics and varied rhythm usually moves the text out of the detected range, but detectors are updated regularly and no result is permanent. If the point is passing a policy check rather than reading well, you are solving the wrong problem.
Why did the same text get different scores on two detectors?
Because they use different models, trained on different data, with different thresholds and different definitions of the score. There is no shared standard. A disagreement between tools is normal and is itself evidence that neither should be relied on alone.