More on the Reading First Evaluation

I was talking with Dick Anderson today. For those who do not know, Dick is an outstanding scholar at the University of Illinois at Urbana-Champaign. He was the director of the Center for Reading during the years when the best research on reading was coming from there. His comprehension research is great.
Dick is not anti-Reading First in my opinion, but I think it is fair to say he isn’t exactly a big Reading First fan. He isn’t against phonics, but tends to think Reading First makes too much of phonics. He feels the same about fluency and phonemic awareness (and, frankly, anything in the curriculum that he feels is not intellectually engaging).
Dick was arguing that Reading First was not evaluated properly. He did not feel that the “regression discontinuity” design used in that study provided an adequate or appropriate test of the effectiveness of Reading First. I’m not a statistician and have wondered in this space whether there are problems with that design. Dick thinks that it reduces the variance in outcomes and reduces the chances of finding a difference.
He might be right, as I say, I’m not a statistician. I still think the bigger problems are these: (1) Reading First (and lots of related policies and information) “contaminated” the control group schools by making their reading programs much more like Reading First than they once were—making it an effective policy, but queering the research study; (2) Reading First schools have such high mobility rates that it is impossible to study them longitudinally. My claim in that last point is not that the Reading First schools are incomparable to the non-Reading First schools because they have such high student and teacher movement (the other high poverty schools suffer from that kind of mobility problem). But any reading intervention aimed at poor schools somehow has to work within the confines of such high mobility.
Of course, Lauren Resnick, the head of the Learning Research and Development Center at the University of Pittsburgh, points out that mathematics improvement has taken place over the past 20 years in the U.S., including in schools like those. The big difference being that learning math across languages is not as challenging as learning reading across those differences.
Language differences and high mobility in poverty schools are not excuses for why Reading First didn’t do as well as it needs to in research. But they are issues that must be addressed adequately in any Reading First redesign or that policy will fail too, if the goal is higher reading achievement for poor kids.