Writing style analysis of IETF Internet-Drafts. Method and limitations →
consensus score across the corpus (click a bar for a random draft in range)
share of typical / mixed / atypical per model · mean deviation % at right
% of drafts where two models share the same verdict
one row per WG or topic tag · box = IQR, line = median, dots = drafts · groups with ≥ 8 drafts
A high median doesn't mean those drafts are AI-written; the ensemble just finds the prose uniform, abstract, and framework-shaped. Same patterns appear in non-native English and formal academic style. Corpus is individual submissions only (no draft-ietf-* WG drafts).
datatracker.ietf.org and sample the first five numbered sections, capped at 6,000 characters.typical / mixed / atypical), returns a deviation % and three evidence bullets.mixed); deviation % is the mean across models.The five dimensions, each 0 (typical IETF) to 10 (atypical):
The prompt tells each model that typos, non-native English, TBD markers, and wire-format tables are human signals; CDDL, ASN.1, and test vectors are strong human signals.
Computed on the full text after lowercasing and stripping non-alphabetic characters, then passed to each LLM as evidence. Only hedging density and average word length correlate meaningfully with the ensemble score in this corpus (|r| ≈ 0.3–0.4). The rest provide context.
| Metric | Formula | What it measures · threshold |
|---|---|---|
| Sentence length variance | σ = √( Σ(Li − μ)² / N ) |
Population std. dev. of sentence lengths in words. Split on [.!?]+\s, sentences < 4 words dropped. Human prose > 8; < 5 = AI-uniform. Commonly called "burstiness" in AI-detection tooling. |
| Guiraud's index | G = V / √N |
Length-corrected vocabulary richness (Guiraud 1954). V = unique types, N = token count. Varied prose > 10; repetitive < 7. |
| Hapax ratio | H = |{w : freq(w)=1}| / V |
Fraction of the vocabulary appearing exactly once. Human prose typically ~0.5; low values indicate repetition. |
| Average word length | μw = Σ |wi| / N |
Mean character length. AI prose skews long (> 6.0); human IETF writing mixes acronyms and technical terms. r = 0.41 with the ensemble AI% in this corpus. |
| Function-word ratio | F = count(closed-class) / N |
Share of a fixed 53-word function-word list (the, of, and, is, …). More function words = more complex argument structure = human signal. < 0.24 leans AI. |
| Sentence-start entropy | H = −Σ p(s) log₂ p(s) |
Shannon entropy over the first word of each sentence. Higher = more varied openings. < 2.0 is a strong AI signal. |
| Hedging density | D = 1000 · count(hedge) / N |
Per-1,000-word count of a 29-phrase list of AI-tells ("leveraging", "comprehensive", "seamless", …). The strongest separator of AI-leaning vs. human drafts in this corpus. |
| Zipf exponent | α from log f = c − α · log r |
OLS slope of log word-frequency on log rank. Natural language ≈ 1.0; deviations > ±0.4 are a weak AI signal. |
| Paragraph length variance | σp = √( Σ(Pi − μ)² / Np ) |
Same formula as sentence variance, applied to paragraphs. Split on blank lines, < 6 words filtered. |
| Self-perplexity proxy | ℓ = (1/N) · Σ ln(freq(wi) / N) |
Mean natural-log probability of each token under a unigram model fit from the document itself. More negative = less predictable = more human. |
| Vocabulary-consistency σ | σ of TTR over 500-word sliding windows (½-overlap) |
Std. dev. of type-token ratio across the document. Human writing shifts vocabulary density between sections; AI output is uniform. < 0.01 = suspiciously flat. |
A detection-confidence bucket (low / medium / high) is derived from normalised unigram entropy: low-entropy, formulaic text is provably hard to detect, so the ensemble defaults to "typical" when confidence is low.
Further reading. These are traditional stylometry features (Guiraud 1954; Shannon 1948) combined with AI-text heuristics. The papers to read on why this is hard: Sadasivan et al. (2023) on detection upper bounds, Kirchenbauer et al. (2023) on low-entropy undetectability, Liang et al. (2023) on bias against non-native English, and Liang et al. (2024) on AI-idiom frequency shifts in academic writing.
Each dot is a model's score on that dimension (0 = human, 10 = AI). Range bar shows how far apart models are; larger spread means more disagreement.
pXX = percentile within the 1,473-draft reference corpus. Green = characteristic of human prose · yellow = ambiguous · red = AI-uniform. Only hedging density is meaningfully discriminative; others are reported for reference.