Analysis and Extension of Sam Shah's 'Best Test Question Ever' (a post about a post)

My friend Sam J. Shah recently blogged about his best test question ever, so it was entirely predictable that I would follow along in lockstep as I so often do, stealing and adapting his best ideas in my own teaching practice. What was less foreseeable was that I might actually inquire into and extend what I see working inside this technique and analyze why it works so well. In doing so, my hope is that I (and perhaps others) can get more mileage out of Sam's brilliance.

Based on an idea from another amazing post by fellow Klingon and expatriate math teacher Bowman Dickson, Sam got the idea to order a set of scratch-off stickers and integrate them into a calculus test question. His objective was to address the dysfunctional reaction and misconceptions that many students have whenever their new and emerging understanding gets challenged.

This dysfunctional reaction is a basically a defense mechanism of the unconscious -- formed in response to years of experience in the massively dysfunctional social system of the math classroom. It conditions students, when in doubt, to abandon their own good sense and everything they know in their life outside the math classroom in favor of half-remembered algorithms and recipes that are all too often are at odds with their own emerging reasoning skills and common sense. 

But they seem to the unconfident student to be what the teacher is asking for.

Sam's test question had two parts: a first part which asked students to use what they know and have learned to solve a problem of the type they'd been focusing on. There was a sticker covering up something important in the information in Part A of the question, but it had no bearing on the mathematical task being set for them, and so they dutifully ignored the hidden information and simply trusted their own minds to solve the problem.

The genius part comes in Part B, where Sam instructed the students to scratch off the sticker, reveal the hidden information, and evaluate their Part A answer in light of the new information revealed in Part B.

What's genius about the Part B question is that this is an emotional question -- not a mathematical one. By challenging readers to defend or retreat from their previously worked solution, the physical action and the question conspire to introduce fear, uncertainty, and doubt into the reader's process of reflecting on their solution to the question in Part A. And as anyone who has done time in the marketing or political strategy realms knows very well, Fear, Uncertainty, and Doubt are three pillars of influencing consumer (or voter) behavior.

When faced with this kind of challenge, the confident student will furrow her or his brow for a moment, then say, "Not so fast, Mr. Shah. I see what you are trying to do here but it won't work. Not for me. I stand by my answer in Part A and I push back against your clever little ploy to knock me off balance."

The interesting process is what happens when the unconfident student is challenged to reconsider her or his answer to Part A. I say this both as a teacher and as an unconfident student who has spent a lifetime learning how to work with my defense mechanisms when I begin to doubt myself or when my work is challenged. This student's thought process goes more like, "Oh crap. Mr. Shah wouldn't have gone to the trouble to put this sticker here and have me scratch it off to reveal a hidden answer unless he *knew* I would mess up Part A and need to rethink the whole thing. He understands this stuff much better than I do, so there must be a good reason for me to back away from whatever work I did in Part A and figure out how to retrofit my solution to fit the new information just revealed."

It is important and useful to address the emotional reaction of panic and self-doubt here -- not simply the mathematical misconception that reveals the underlying panic and self-doubt. In this case, the misconception is a symptom, not an underlying cause itself.

If we are going to help students develop the courage to trust their own emerging skills and understanding, we need to help them learn how to (a) notice their panic in this kind of situation and (b) address it in a more effective and non-automatic, non-self-abandoning way. As Rudolf Dreikurs would say, we need to encourage our students' courage. And the only sustainable, durable first step that will help them to overcome their unconscious fears is to help them tap into their body-based awareness that they are freaking out.

And this is where Sam has offered a powerful yet simple means to do that -- using only a scratch-off sticker and a thoughtful question.

Once they have noticed that they are freaking out, students have at least 100% more options for next steps in dealing with their mathematical understanding or its lack.

But to move through the emotional blockage, a student first has to notice that the blockage is there.

And so, Sam, this is why I am, as always, your biggest fan.