Can You Spot AI Writing in the Wild?
A challenge for teachers and professors.
Teachers everywhere are grappling with a new reality: students turning to AI to help them complete their assignments.
Many of us think we’ve gotten good at spotting AI-generated text, catching the clues like awkward wording, overly sophisticated vocabulary, or repetitive punctuation choices — em dash, anyone?
But how accurate are our internal AI detectors, really?
To put your skills to the test, I’ve partnered with my brother (a high school English teacher) to gather some real, verifiable student writing, and we are putting it head-to-head against AI generated passages.
Your challenge? Read through each of the student writing samples below and decide if it was written by a real human student, or if it is a fake.
Putting your AI detector to the test
Below you will find four prompts. Each one has a roughly 100-word student response. After reading the response, decide if you think it is real or fake and commit your answer to the poll. You’ll find the correct answers at the very end of the post (no peaking!)
Prompt 1: Discuss how technology has impacted society so far this century, mentioning both positive and negative effects.
Student A: In this twenty-first century, technology has taken society by a chokehold. Televisions are everywhere, computers are used in almost every school, and almost every American owns a cell phone. The usage of these devices isn’t bad, but they aren’t good either. They constantly call people to look at them, to click the tab, to win rewards, and to waste hours staring at the device. Technology, cell phones specifically, are great tools to use, but they can quickly devolve into an addiction or bad habit. With gentle reminders and self-control, people can return cell phones to their original purpose: a tool to be used.
Prompt 2: Describe a personal challenge you faced during high school and explain how it affected you and how you coped.
Student B: My anxiety started to spike with the start of high school. The year before, I experienced two significant losses: first, my dog met an unexpected and violent death. Second, my only surviving grandma suddenly passed away after being diagnosed with Acute Myeloid Leukemia. After these two significant changes in my life, I stumbled my way through the beginning of high school, trying to catch my bearings. I was able to make it through the day, but I was unable to pinpoint why I was always so scared. Eventually, with the help of my family, I realized that what I felt would become my new companion: anxiety.
Prompt 3: Describe an experience that taught you the importance of empathy or kindness.
Student C: In sophomore year, there was a new girl whose locker was next to mine, and she mostly kept to herself. Everyone just assumed she was shy or didn’t want friends. One morning, we both got to our lockers early, and at first it was awkward since we'd never really talked before. But we started chatting, and she mentioned she'd moved schools twice that year and didn't know anyone here yet. I realized how lonely that must feel, so I invited her to hang out at lunch. Seeing her face light up showed me how important it is to notice people who might feel invisible.
Prompt 4: Describe an activity or interest that you pursued even though it was challenging. How did it shape your character?
Student D: I started learning guitar in freshman year and, honestly, it was a lot more difficult than I expected. At first, my fingers were sliding off the strings, and every chord sounded terrible. I was about to give up, especially when my friends seemed to learn it quicker. But I practiced every night, sometimes until my fingers hurt. Finally, after months, songs started to sound like songs, and that was incredible. Guitar playing taught me patience and frustration tolerance. When anything seems like it's impossible, I remember how lost I was when I started out and realize that I just need to practice some more until I get better.
Behind the scenes: Making AI writing feel human
Did you find this more challenging than you expected? I hope so… (otherwise, it would be a pretty boring post, I imagine.)
To generate realistic responses from ChatGPT, I gave it specific instructions to imitate a high school student and avoid the typical hallmarks of AI authorship.
I took the further step of passing the result through an AI “humanizer,” which added imperfections to the writing. (If that strikes you as unfair, please know that I got the idea to use the humanizer from actual high school students.)
While the default output from a tool like ChatGPT often has characteristic hallmarks that make it easier to detect, it is important to remember that just a little bit of additional massaging can achieve results that are indistinguishable from that of human writers.
How did you do?
Ok, now it’s time to check your work. Here are the correct answers for each student.
Student A: Real
Student B: Real
Student C: Fake
Student D: Fake
I’m curious to hear how you did. Were your surprised by your results? You can leave a public comment or privately reply to this email to let me know.