Monday, March 13, 2006


SAT Scoring Problems and The Three Wyman Questions

The recently reported problems with erroneous scores on SATs (see SAT Errors Raise New Qualms About Testing) should raise more than just qualms. One has to call into question the whole issue of quality control at Pearson Educational Measurement. It seems this incident offers a perfect opportunity to explore The Three Wyman Questions.

The Three Wyman Questions are a methodology which I have developed for effectively dealing with service problems. When a service anomaly occurs, I have found that unless all three questions are answered, you are leaving yourself open for further service problems. These questions are all obvious, but you would be surprised at the number of times they aren't asked.

So what are the questions?

1. What went wrong?
This is the most obvious question and more often than not it does get asked (and answered). However, I'm sure we've all encountered the situation where a computer system hung and the remedy was to restart the system. If and when the question gets asked, the response is generally "I don't know, but restarting the system fixed the problem." Wrong answer. The problem has not been fixed. In Pearson's case they appeared to identify excess moisture and light marks as the problem. So far, so good.

2. How do we prevent the problem from happening again?
Obviously, unless question one was answered, it makes no sense to attempt to answer this question. Assuming that this is not the first time this problem has occurred for Pearson, they obviously didn't answer it the first time around. While the full details of the Pearson problem are not clear, one naive answer to this question could have been "We'll add registration marks to the form to be scanned and if the marks are not found where expected, we'll assume the form did not get scanned properly."

3. Should the problem recur, how can we detect it before our customer does?
It's never a good idea to expect your customers to do quality assurance for you. First of all, they are extremely fickle. They'll just as soon tell their friends how bad your service is as opposed to letting you know. Also you'll look very stupid when your customer tells you about a problem you don't know about. Frequently, a service provider will assume that as long as question two has been addressed, the problem has been resolved and will never occur again. This is a fatal mistake. There's no such word as "never" when it comes to problems recurring.

By now, you are saying that all this is nothing more than common sense. Of course it is. But please tell me why these questions go unaddressed so frequently?

Why don't SAT grading companies like Pearson ask these questions of themselves?

Well, being in a monopoly means never having to say you're sorry.
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?