Framing Matters: Teaching and Assignment Design In The Age of ChatGPT Part 2

In my last post, I talked about how I explored what ChatGPT could and could not do with the assignments I give my students. Now I want to talk about another little “experiment” I did with my students themselves that showed so much about how we receive AI and what we perceive as good writing.

This time, I had ChatGPT respond to a writing prompt I often use with students. I told it “write a paragraph that begins ‘I am a writer because.'” I love this prompt with students because it forces them to think about ways that they do, in fact, write. James Paul Gee argues that learning can fail when there is a conflict between some part of the learner’s identity and the content being learned, such as how we know that girls score poorly on math tests after being exposed to stereotypes that suggest that being a girl somehow means they’re not good at math. By forcing students to make some kind of statement about being a writer, I’m trying to help them negotiate that kind of conflict in their identity if it exists, which for many students it does. Obviously, though, ChatGPT does not require any such mediation, as its learning algorithms function in a very different way.

At any rate, this is the paragraph that it gave me in responses to “write a paragraph that begins ‘I am a writer because'”:

I am a writer because words are the brushstrokes with which I paint the canvas of my imagination. Through the dance of letters on a blank page, I weave worlds that exist only in the realms of thought and emotion. Writing is my sanctuary, a realm where I can articulate the unspoken, give voice to the silent, and distill the chaos of existence into the elegant simplicity of prose. It is through the alchemy of language that I find solace, purpose, and the boundless freedom to explore the vast landscapes of human experience. As a writer, I navigate the labyrinth of ideas, embracing the power of words to transcend time and connect the threads of disparate souls. In the written word, I discover not only the essence of others but also the reflection of my own identity, making the journey of writing an odyssey of self-discovery and a celebration of the shared human narrative.

generated by Chat

It is, honestly, word vomit. It’s fluff. It’s meaningless. I hate it when students try to write like this. It is, to quote the Bard, “sound and fury, signifying nothing.”

With my first class, I mentioned my little games with ChatGPT and told them that it generated terrible writing. They asked to see it. I showed them. They agreed univocally that it was terrible writing and laughed at it with me. Of course they did: here was the teacher telling them that this was bad writing. And the teacher is the authority.

With the second class, I put two paragraphs side by side. Both are imaginary in a way—that is, they do not represent any real human’s feelings on their writerly identity. For the second paragraph, I quickly wrote a version in the persona of my imaginary student Justin Bailey, who is designed to be a pretty average B-level student who frequently gets stuck in their writing and needs a little help to get unstuck. So they saw this:

I am a writer because I often have to be. I don’t think I’m a great writer, but I do write a lot, so I’m a writer. I write because I have to for classes at school and just to do other things, too. For instance, I write reviews of products I buy. I write text messages to my friends. We have written conversations on Discord about the games we’re playing and try to persuade each other to play new games or to try a new strategy. In school, I have to write essays and paragraphs and free-response questions on tests. In all these cases, I try to get better at writing all the time because I need to communicate my ideas clearly. So, I am a writer because I have to be, and I’m always trying to get better.

Sample paragraph I wrote in the persona of “Justin Bailey,” an imaginary student.

In this case, I did not tell them which was which. Instead, I asked them the question “Which is better written?” Most of them chose the ChatGPT paragraph. Then I asked the question: “Which would you rather read?” Most of them chose Justin Bailey’s paragraph. “Then isn’t that the better written one?” I asked. This, of course, led them to the same conclusion that I had upon reading ChatGPT’s work (indeed, one student even noted it looked a bit suspiciously computer generated), and I explained the whole game and that ChatGPT had written the first paragraph. We then had a lovely conversation about when to use AI and when it would be unethical or ineffective to do so—these are absolutely the conversations we need to be having with our students.

In the third class, I asked them “Which would you give a higher grade to?” which seems to ask “Which is better written?” but is actually a very different question. In this case, almost all of them chose the AI generated paragraph–all but two students, in fact!

To the two students who said a lower grade, I had a conversation with them and discovered that they were pretty aware that the AI generated paragraph was, at best, very fluffy. I let them in on the secret, and they were entirely unsurprised.

Then, I conferenced with the group that consisted of most of the class. I asked them about their reasons, and they pointed to the elevated vocabulary and insisted that this “person” seemed to be very “passionate”. I asked them to summarize the paragraph and explain the “person’s” relationship to writing, and they struggled. Then, I gave them a choice: “If you want to change your mind, you can go to the other group now.” Half the group left. Now the classroom was more evenly divided.

I had them debate a little longer before revealing that it was, in fact, AI. I pointed out some of the tells–lots of very fancy language and little content–and explained how AI generates text by looking for patterns in a large database.

I explained why I would grade the human-generated one better even without the interference of AI and its ethical quandaries. Simply put, the human-generated one is more specific and has clearer meaning.

So what was the take-away from this lesson, taught in three different ways?

The first thing is that students will tend to adopt the values the teacher expresses, so we have to be careful about how we frame readings. If we tell them it’s good writing, they’ll believe us.

The second is that, absent the teacher framing something as good writing, the students are generally under the belief that good writing is writing that sounds fancy and uses complex sentences and long words. That means somewhere along the line, we have failed to teach them that good writing is communicative, not showy. But we can’t blame ourselves, or the students, too much for thinking that good writing is fancy writing. It’s a common misconception in our culture. We show the “smart” characters by having them talk like AI, evidently–or perhaps AI talks like our “smart” characters.

And maybe it’s a symptom of our author-centric and grade-centric language around writing. We teach high schoolers that the purpose of reading is to serve the writer by guessing at their true meanings and intentions. We teach students to compete with each other to show off their vocabularies and we test them on standardized tests by seeing how many Latin and Greek root words they can define.

But in the age of AI, we need to be careful how we frame what is meritorious in writing. AI does fancy vocab, high word counts, and other markers we often test students on well. Perhaps even undetectably well in some cases. But if we want to make sure we value human communication, we need to frame the purpose of writing as establishing relationships, something AI really can’t do yet.