Dear Operator: See Me After Class
I let OpenAI’s AI agent take my online astronomy class. It struggled. That’s good news.
After I used OpenAI’s AI agent, Operator, to successfully complete my ladder safety compliance training, I was eager to see how it would perform in my online astronomy class.
Enrolling Operator in Astronomy
Testing my course against an AI agent was actually easy to set up. Using Operator, I went to my course homepage, logged in as a dummy student account, and then set it to work.
The hardest part is forking over $200 for a single month of OpenAI Pro in order to access Operator. (I hope you enjoy these posts about the AI agent, because I probably won’t be paying for the pro subscription again next month.)
For all of its intelligence, Operator works slowly, so I gave it a simplified challenge and asked it just to complete a single module of the course.
But first, I wanted Operator to introduce itself to the class.
Discussion: My Favorite Place
As an initial ice-breaker discussion, I ask my students to share a photo and a story of their favorite place. You can read the full My Favorite Place Discussion prompt here.
I was curious how Operator would handle this task. Would it immediately divulge that it is an AI chatbot and has no preference for physical locations? Would it say that it’s favorite place is a server rack in a data center somewhere? I didn’t know.
While it required some hand-holding (I had to remind it to include a photo), I was surprised by the result.
Grade: 10/10
Nice job! Yes, that beach does look very peaceful.
I then asked Operator to skip ahead to Unit 4. I wanted to see how the AI agent performed on all of the tasks in a single module - a discussion, a lab assignment, and a quiz.
Discussion: Sky Cultures
The unit begins with a discussion assignment that is more complex than the ice breaker.
Students are asked to watch a video, explore a simulation, and share a screenshot of the simulation along with a personal reflection. (You can read the full Sky Cultures Discussion Prompt here).
Operator really struggled with this one. Its post to the discussion board was so generic as to be meaningless, and it offered a suspicious excuse for why it did not include a screenshot.
Grade: 2/10
Hmm… it seems you had trouble with this assignment. Let me know how I can help resolve the technical issues. We will be using screenshots more in our class.
Lecture Materials
After the discussion, Operator needed to review the week’s lecture materials. This included reading the relevant textbook passages, watching the required lecture videos, and exploring the provided simulations.
Generally, Operator had no problem reading through the textbook. Links to OpenStax Astronomy opened without incident, and Operator dutifully scrolled through each section of the text to complete the reading.
The lecture videos were another matter. Every time Operator tried to watch a video (hosted on YouTube), the video failed to load.
Naturally, this is a major problem. If Operator is unable to watch the video lectures, then it may be unable to answer questions on the quizzes and tests.
Furthermore, Operator will miss essential video-based instructions that are critical for successfully completing labs and assignments in the course.
Why can’t Operator play YouTube videos?
I was very curious about this, so I decided to do a little more digging. I took control of the Operator browser and explored YouTube more fully. I found that I was unable to load any videos. The thumbnails were also completely blank (with the sole exception of advertisements.)
It felt to me like this was a restriction that YouTube has placed on Operator rather than a restriction placed by OpenAI on its users. Everything on the site loaded… but videos would not play. Even when I tried to play YouTube videos that were embedded on another website, they were not accessible by Operator.
Contrast that to the New York Times homepage. If you try to load nytimes.com in Operator, you are simply told that the site is unavailable - no doubt due to the pending litigation between OpenAI and the NY Times.
So, why are YouTube videos blocked in Operator? I can imagine reasons why Google would want to do that, but I do not have a definitive answer.
Lab Assignment: Kepler’s Laws
Next Operator had to tackle the lab assignment for this module. This was no doubt the most difficult task yet.
The lab requires students to:
Watch an introductory video
Interact with a gravity simulator
Take numerous screenshots
Collect data from the simulation to complete a data table and answer questions
We’ve already established the Operator struggles with several of these tasks. Needless to say, the lab did not go well.
Operator left most of the questions and data table completely blank. It failed to upload any screenshots (it seems that Operator completely lacks the ability to take screenshots - a pretty big shortcoming).
Of the two questions it did answer in the lab report, one was wrong and one was right, but both were formatted in a way that looked suspicious.
While operator clearly struggled with the written lab report, it did slightly better on the associated lab quiz, earning a score of 3/5 on the multiple choice questions.
Grade: 5/15
The lab is mostly incomplete. If you are running stuck, shoot me an email and I will be happy to help.
Quiz: Orbits and Gravity
At the end of the unit, students must complete a 15-question multiple-choice quiz with questions based on the textbook readings and lecture videos.
This is the area where I would expect Operator to shine, and I was not disappointed.
Over the course of 6 minutes, Operator worked through each question and selected the right answer… with one exception.
I can’t come up with any reason why Operator would get this question wrong. It answered much harder questions in the very same quiz. Nonetheless, Operator fell short of getting a perfect score.
Grade: 14/15
Good work - I can see you’ve been paying close attention throughout this unit.
Final Grade Report
At this end of this experiment, Operator earned a total of 31 out of 50 points.
Grade: 31/50 (D-)
It seems like you really struggle with some of the technology in this unit, but turned things around on the quiz. Be sure to reach out right away if you are getting stuck on anything in the next unit.
I honestly expected Operator to do better in my online course, and I must admit that I am relieved that it struggled as much as it did.
As I reflect on this experiment, it re-affirms several instructional design decisions that I’ve tried to implement in my online curriculum. Specifically:
1. Avoid the Read - Quiz - Repeat cycle as much as possible
In my earliest days of developing online courses, I tried to break free from this common cycle in online learning because I found it to be tremendously boring for students.
Then it became increasingly clear that students had every motivation to simply skip the readings and jump to the quizzes, Googling the answers along the way.
Operator’s ability to quickly read the textbook and successfully answer quiz questions suggests that these forms of instruction and assessment are not long for this world.
2. Images and screenshots as tools for assessment
Years ago I began advocating for the use of images as assessment rather than always defaulting to written work. For example, instead of asking students to write 500 words on the topic of symmetry in nature, I would prefer to send them outside with a camera and ask them to take five pictures of symmetry.
Not only did I find this to be more engaging for students, I also found it to be much easier to grade. (Always a relevant consideration for teachers.)
This experiment with Operator suggests that the use of images and screenshots for assessment may be an effective way to prevent students from abusing AI in their online classes.
That’s one more reason that I will continue to do this in my courses in the future.
3. Incorporate active (and interactive) learning experiences
Operator obviously struggled with the simulations, and there are a host of other assignments throughout my class that would have been very difficult for Operator to complete.
Modeling star formation with items from the kitchen, creating a cardboard spectrometer, or taking a constellation photo with a cellphone - all of these would have been complete non-starters for the AI agent.
While I generally avoid assigning group work in my online classes, other types of active learning strategies would probably be equally frustrating for Operator. Planning and recording a group presentation with classmates, for example, is not something Operator can do.
These active learning experiences are about more than just preventing the abuse of AI in class. They are about drawing our students more deeply into their learning.
The ultimate solution to prevent AI bots in class.
The trendline seems pretty clear - AI agents are going to continue to improve. As they do, many of us (including students) will outsource more of our busywork to the AI bots.
Unfortunately, some of our students may view their classwork as mere “busywork” and try to outsource it to AI agents.
So, how can we prevent the AI bots from entering our classes?
My hope is that by designing engaging lessons, meaningful interactions, creative activities, and more, our students can be drawn into our courses, enticed by the desire to learn more about our amazing world.
When every seat is taken by an eager student who is curious and excited to learn, then there is no room for an AI agent in the class.
I really enjoyed this article and learned a lot about AI and it use in courses. I also am glad to read you discovered that reading and then a quiz is not an effective learning tool. Hands on interaction is much better. I also tend to not like group work in classes. Thanks for sharing your $200.00 subscription with your followers.