The mediocre piece of fiction discussed by Vauhini Vara—a writer herself, but mostly a journalist for The Atlantic, The New Yorker, and The New York Times Magazine—might as well be a fraud for being written with the help of an LLM, but this fact was not proven and cannot be proven—at least, not by using Pangram. Furthermore, she contradicts herself regarding the false-positive rate for English-language writing by non-native English speakers. I will explain.

A Pangram lover loves to expose fraudsters

On May 21, Vauhini Vara wrote this piece in The Atlantic: This Literary AI Scandal Changes Everything. It was about “an all-but-unknown Trinidadian writer named Jamir Nazir” and his story that was one of five regional winners of the Commonwealth Short Story Prize.

On October 11, she persevered: Yet Another Literary Prizewinner Seems to Be AI. This time, the Zimbabwean writer Simbarashe Steyn Kundizeza is accused of winning the prestigious Caine Prize for African Writing with a story written with AI help—or maybe the story was entirely AI-generated?

To paraphrase the greatest living genius, “I never heard of the prestigious Caine Prize, and nobody did.” But I noticed a strong disconnect between the “seems to be” in the title and the article’s harsh bashing. Someone should be sued for defamation.

First, let me acknowledge that the story is so much “meh” and so little worth a literary prize! You can read it here: The Blue Fighter.

The problem starts right here:

The AI detector Pangram flags almost the entire story as AI-generated; a notable exception is its epigraph, also about a father with a fish fixation, which is excerpted from what’s described as an unpublished poem by Vangile Gantsho. This isn’t the first prize-winning story to be flagged by Pangram.

Let me put it as boldly as I can:

In truth, Pangram cannot prove anything! It’s not just unreliable; it literally cannot produce any proof that would stand up to legal scrutiny! A plagiarism detector shows clear evidence, but when an AI tool “believes” something is AI-generated, where is the proof? Everyone whose work has been rejected “because Pangram said it’s AI” should sue the organization that made the claim and win in court!

This Vara person admits that AI detection is fallible and links to an article by Matteo Wong: America Has a Pangram Problem. Nonetheless, she has a job to do, which is to expose fraudsters, regardless of the reliability of the tools used for that purpose:

But Pangram, which uses AI for its classification of text, is highly accurate: The company says its tool mistakenly labels human text as AI in about one in 24,000 cases; outside research also finds false positives to be extremely rare. When I ran all 25 of the previous Caine Prize-winning stories through Pangram, it labeled them as 100 percent human-written; the same is true of Kundizeza’s “The Whistleblower.”

Oh, my. Everyone else says Pangram isn’t very accurate, except the makers of Pangram and the retards who pay to use it as a modern “lie detector.” They and Vauhini Vara.

For the purpose of this blog post, I really don’t care who (or what) wrote that piece. If people who frequently use ChatGPT for writing tasks are indeed accurate and robust detectors of AI-generated text, they should serve on juries that give such awards and vote against a suspicious finalist. Instead, this horrendous person decided to act as a Soviet public prosecutor. Heck, even I don’t trash people without proof of their misdeeds!

Surely, Pangram must be more sophisticated than using crap criteria like the use of the “not x, but y” sentence pattern, em dashes, or sophisticated language that most people on Earth, being uncultured and semi-oligophrenic, would never use. I’ve always used “not x, but y,” and for years I’ve used the em dash—without spacing, mind you!—as part of my attempts to follow AmE rules, despite being taught BrE in school. Just the other day, I was stunned to discover that Gemini answered in my style, not the other way around. Either I was born a chatbot, or Gemini started to mimic me. Once again: not the other way around! If this blog’s style has worsened since the advent of chatbots, it’s only because I was forced to write about such crap! Unlike most people who use ChatGPT and Claude (I used about 20 different chatbots), I never asked any chatbot to write any text for me! (Occasional translations of an idiom don’t count.)

Our Soviet prosecutor has her own preferred “proofs”:

What I also noticed, more disturbingly, was the presence of stereotypes about African life common among outsiders: a blundering, unemployed father; a resilient matriarch who dispenses clichéd approximations of wisdom; an environment of deprivation. Research consistently shows that AI products reproduce cultural tropes, including those having to do with the African continent.

Oh, research shows that A → B; therefore, if B, then A. Marvelous logic. Any and all cultural tropes must be AI-generated, eh?

The prosecutor’s arguments change with the season

Here’s an argument from this recent article:

Pangram’s internal research finds false positives to be rare or nonexistent across multiple races and languages, as well as for English-language learners writing in English. In a separate peer-reviewed study, researchers at the Vrije Universiteit Brussel found zero false positives for English-language writing by non-native English speakers.

And here’s what he wrote back in May:

A much-cited Stanford paper published three years ago found that AI detectors had a higher false-positive rate for text written by non-native-English writers than they did for text from native English authors.

Let’s put it in simpler terms:

  1. For English-language texts written by non-native English speakers, Pangram has zero false positives.
  2. For English-language texts written by non-native English speakers, AI detectors have higher false-positive rates than for texts written by native English writers.

The only logical conclusion that could be drawn from the above assertions is that Pangram is not an AI detector. This could be true if we agree that Pangram is a fraud. Otherwise, as a non-native English speaker, I’d rather believe that AI detectors, crappy as they are, can be even crappier when analyzing non-native English texts (GPT detectors are biased against non-native English writers).

Pangram isn’t another GPT-powered tool; it’s sophisticated crap designed to persuade clueless people to pay for it when their own judgment is low-grade. That doesn’t make it any less of a fraud, even though it has customers in the Fortune 500, S&P 500, FTSE 100, and the Ivy League.

Just for fun, Grammarly found 3% “possible AI,” which is short for “I burned some tokens, and I try pretending it wasn’t for nothing”:

Quillbot “found” 2%:

Balderdash. All such detectors are plain nonsense.

I still can’t decide whether Vauhini Vaa is a fraud or in desperate need of psychotherapy. Maybe she suffers from PTSD due to some childhood trauma. Or maybe she’s just a bitch, which is not incompatible with a slight mental retardation. Either way, stop reading her shit!