What Does It Mean that a Reading Program Works?

Recently, the What Works Clearinghouse (WWC) posted a report on multi-sensory programs for teaching children with reading disabilities. The report indicated that Orton-Gillingham, Wilson Reading, and other similar programs lacked convincing empirical evidence of their effectiveness. This has set off a lot of angry e-mailing this weekend from those who “know” these methods work.

http://http://ies.ed.gov/ncee/wwc/

  Although I serve as a reading content expert for WWC, I had nothing to do with this report. They only involve me in those aspects of their work when they have a substantive question about reading instruction or assessment, and on this report, I guess there were no such questions for me.

  In any event, I strongly support WWC, though at times it has made choices that I have disagreed with. In my experience, it is rigorously, carefully, and fairly adjudicated and its reviewers make a serious effort to find the research and to evaluate it in a reliable manner.

  I know that it is upsetting to not get good news, but what does it actually mean that WWC says there is no evidence supporting these programs? First, and most important, it does not mean that those programs do not work or cannot work or that you should not use them. “No evidence” means no evidence and nothing more. “No evidence,” one way or the other. You may be able to make such a program work, or you might not, but no one has yet studied it rigorously. (There are many more programs that lack such evidence than there are ones that have it, and let’s face it, plenty of kids learn to read from such programs.) Don’t read too much into a lack of evidence. That is more about the standards we have as a field and how much we are currently willing to invest in programs.

  But what about when the programs that WWC says something does work? I think too often teachers and principals think that is the end of the story. But it is not. When WWC says something works, you have some other questions to answer.

 What does it mean to “work”?

So, when WWC says that there is evidence that a program worked it means that someone conducted at least one well-designed rigorous study showing that a program worked better than… well, better than something – or nothing. There is not a convention dictating what the control group is to be doing in such studies, so it is well worth taking a look at what they were up to so that you can know what the program was better than. (In some phonics studies, for example, control group children are in a conventional reading program without phonics. In other cases, the control group kids are receiving no reading teaching at all. The effects are generally larger when the comparison gets no treatment than when it is some other version of reading instruction.)

  If WWC says the program worked and indicates an effect size, you might figure that the programs with the biggest effect sizes are the ones to buy. That would generally be true if the control conditions were standardized in some way, but that is not the case. Bigger effects are likely to be noted when the comparison is with nothing, than when it is put up against some other effective way of teaching reading.

  To know what it means to have worked, it is also important to know what the outcomes were. WWC is very good about laying out what the specific measures of reading were that the students actually improved upon in the studies, and that is something that you want to know. If you are trying to improve kids’ reading comprehension, buying a program proven to improve only decoding may not be a wise choice, despite the WWC imprimatur.

Who did it work for?

Okay, so now, you know this program worked and you know what it means to have worked, so the issue is, with whom did it work? Many programs are sold on the basis of studies that didn’t treat students like the ones who you teach. Were the kids rich or poor, black or white, English learners or native English speakers, preschool or primary, disabled or abled? If a program works with one group, it might not work as well with another.

When did it work?

Finally, under what conditions did it work? Was this instruction given during the school day or afterschool? Was it supplementary to regular classroom instruction or did it take the place of such instruction? How much of it was needed to make a difference? Who delivered the teaching—regular teachers or the program creators? All of these questions get at whether or not you will be able to make the program work in your school, under the circumstances when you will be using the program. I always think of the district that purchased Read 180 on the basis of its evaluation studies; the program, as studied, delivers 90 minutes a day of instruction. The district purchasing it planned to use it for 45 minutes per day instead. I bet it doesn’t work!

  By all means look at the research evidence on programs in which you are interested. If there is no evidence, it is definitely a strike against the program, but it should not be a fatal strike given that so few programs have such evidence at this point in history. And if there is positive evidence, then you need to look further to be sure it works at what you are trying to do, with the kinds of kids you are teaching, and under circumstances that could be replicated in your school.