Monitoring the Reading Comprehension of Older Students

  • 09 December, 2022
  • 24 Comments

Teacher question:

I’m writing to you about high school progress monitoring for reading comprehension. Our school has learning goals for Reading Comprehension. Every two weeks, students read an on-grade level passage and answer 5 multiple-choice questions that assess literal comprehension and main idea. Our data are not matching well with other data that we have (such as course passing rates and state assessments). What might be a more effective progress monitoring process, that go beyond the literal level, and that would provide information the teachers could use to improve instruction.

RELATED: Shared Reading in the Structured Literacy Era

Shanahan response:

I’m not surprised that approach is not working. There is so much wrong with it.

First, why test students so often? Does anyone really believe (or is there any evidence supporting the idea) that student reading ability is so sensitive to teaching that their reading performance would be measurably changed in any 10-day period. Performance on measures like reading comprehension don’t change that quickly, especially with older students.

I don’t think it would be possible to evaluate reading comprehension more than 2 or 3 times over an entire school year in the hopes of seeing any changes in ability. It is unlikely that students would experience meaningful measurable changes in comprehension ability in shorter time spans. The changes from test-to-test that you might see would likely be meaningless noise – that is test unreliability or student disgust. Acting on such differences (changing placement or curriculum, for instance) would, in most cases, be more disruptive than helpful.

Second, I get why we seek brief, efficient assessments (e.g., a single passage with 5 multiple-choice questions). Let’s not sacrifice a lot of instructional time for testing. We have such dipsticks for monitoring the learning of foundational skills (e.g., decoding, alphabet knowledge) with younger students and it would be great to have something comparable for the older ones too.

Unfortunately, reading comprehension is more complicated than that. To estimate reliably the reading comprehension of older students takes a lot more time, a lot more questions, and a lot more text. That’s why typical standardized tests of reading comprehension usually ask 30-40 questions about multiple texts – and texts longer than the ones that your district is using.

How many questions does a student have to answer correctly to decide he/she is doing well? Remember, guessing is possible with multiple-choice questions, so with only 5, I’d expect kids to, by chance, get 1 or 2 correct, even if they don’t bother to read the passages at all. There is simply no room in that scenario to either decide that the student is doing better or worse than previously or to differentiate across students. If a student got 2 items correct last testing, and this week he gets 3, does that mean he showed progress?

Third, reading comprehension question types are not useful for determining instructional needs. Studies repeatedly find no meaningful differences in comprehension across categories like literal, inferential, or main idea categories. If a text passage is easy for students, they usually can answer any kind of question one might ask about it; and, if a passage is hard (in readability and/or content), students will struggle to answer any of the question types.

That means there is no reason to either limit the questions to literal ones or to shift to a different questioning regime. In fact, doing so might focus teacher attention on trying to improve performance with certain types of questions, rather than on decoding, fluency, vocabulary, syntax, cohesion, text structure, writing, and other abilities that really matter.

Fourth, the measurement of readability or text difficulty is not as specific or reliable as you might think. Look at Lexile levels, one of the better of these tools: texts that Lexiles designate as grade level for high school freshmen are also grade level for students in grades 5-8 and 10-12. This kind of overlap is common with readability estimates, and suggests that passages judged to be 1200L will differ in the difficulties that they actually pose for students. Kids might be more familiar with the vocabulary or content of one text or another which can lead to dramatic outcome differences from assessment to assessment.
That’s why the standardized comprehension tests not only pay attention to readability ratings but evaluate combinations of specific passages to make sure that those combinations are going to provide sufficiently accurate and reliable results.

I would suggest that you test students twice a year (at the beginning of each semester) with a more substantial validated reading test. To monitor more closely how students are performing with what is being taught.

For example, one valuable area of growth in reading comprehension is vocabulary. Keep track of what words are being taught in the remedial program and monitor student retention of these words.

Or, If you are teaching students how to break down sentences to make sense of them, then identify such sentences in the texts students are reading to see how well they can apply what is being taught. The same kind of monitoring is possible with cohesion and text structure.

My point is that since you cannot provide the kind of meaningful close monitoring of general reading comprehension that you would like, instead monitor how well students are doing with the skills and abilities that you are teaching – that should provide you with some useful index of progress.

READ MORE ARTICLE HERE: Shanahan On Literacy's Blogs

Comments

See what others have to say about this topic.

Katie Stewart
Dec 10, 2022 06:50 PM

Hi, Dr. Shanahan!

I really enjoyed this post. It reinforces some of my own experiences and beliefs, but my question is about how to square this with the requirements of MTSS and IEPs for progress monitoring. At our middle school, we are being told give all tier II students a comprehension goal and to use Reading Plus as the intervention and progress monitoring tool. We must provide a score 2x a month. Of course, there are a lot of problems with this, including the fact that comprehension may not be the most important goal for these children. If we aren't using the monthly measures for gauging progress in comprehension via a passage and questions, how do we show progress as required by the state? Or do we jump through the hoops and rely more on data from measures like you suggest? We do give the NWEA MAP assessment 3x a year...

Thank you!

Timothy Shanahan
Dec 10, 2022 06:52 PM

Katie-

It is miserable when the folks in charge require information that can't validly be provided. I would suggest using the kinds of measures that I have suggested.

tim

Chase Goff
Dec 10, 2022 07:04 PM

Thank you for the post. This makes me think about reading in general at the secondary level. This is one of the hardest things to get my mind around because we are balancing student needs with state requirements, such as MTSS, IEP’s, graduation requirements, credits, etc. And as you have pointed out, many of the tools we use to measure needs aren’t necessarily aligned with what students actually need.

Could you point me in the direction of research on the best ways to provide services to struggling readers at the middle and high school level?

Tiffany
Dec 10, 2022 08:33 PM

You mention older students, but are these claims also true (around comprehension) for younger readers, such as intermediate elementary? Would they show more growth in shorter time frames because of the growth in the foundational skills?

Timothy Shanahan
Dec 10, 2022 09:08 PM

Fair point, Tiffany. Most of those problems are inherent in the evaluation of comprehension. Growth rates are definitely much faster and easier to measure with younger students, but it is still impossible to measure all of those individual standards or skills that many schools aim at even with younger children.

tim

Timothy Shanahan
Dec 10, 2022 09:14 PM

Chase-

There isn't a whole lot, but these research studies should provide some direction.

Vaughn, S., Martinez, L. R., Williams, K. J., Miciak, J., Fall, A-M. & Roberts, G. (2018). Efficacy of a high school extensive reading intervention for English learners with reading difficulties. Journal of Educational Psychology, Advanced online pub. doi:10.1037/edu0000289.

Vaughn, S., Roberts, G., Wexler, J., Vaugn, M., Fall, A.-M. & Schnakenberg, J. (2015). High school students with reading comprehension difficulties: Results of a randomized control trial of a two-year reading intervention. Journal of Learning Disabilities, 48(5), 546–558. doi:10.1177/0022219413515511.

Tim

Jonathan
Dec 10, 2022 10:53 PM

I work in a UK primary school. What would you say were the issues with the Reading SATs? Similar to what you wrote above? What would you suggest as an alternative method of schools assessing comprehension? Should teachers instead be assessing how well students apply strategies, retention of vocabulary taught etc?

Jeffrey Hellmers
Dec 11, 2022 01:49 AM

Dr. Shanahan,

Thank you so much for this excellent post. It was eye-opening for me on a question that has bothered me for years. Like the teacher who sent you the original question, I used to give weekly multiple-choice quizzes based on a one-page reading story. (I’m a fifth grade teacher.) These were part of our adopted curriculum. But I always struggled to know what to do with this information! Okay, so one of my students only answered 2 questions right out of 5. I know this person is a struggling reader, but there’s no useful information here. (I gave up these tests last year. Thankfully I work in a school that gives teachers lots of flexibility.)

So your point about the complexity of measuring reading comprehension is very helpful. Now I see that rather than trying to measure one massive and complicated thing called “comprehension” I should focus instead on giving assessments the results of which will actually be able to guide my teaching. If I’m teaching them how to make complex sentences using subordinating conjunctions, I can measure whether they’re getting it. If not, I will be able to clearly see in their answers what the issues are and I will be able to re-teach the concept or help the students who need it individually. Or if they’re learning about pronouns, I can ask specific questions about pronoun antecedents to see if they really know what “it” refers to in a given sentence.

In other words, there’s no point in assessments that don’t give you information that can change your teaching and student learning.

Thank you!

And thanks to your many readers, who post excellent comments. I have been reading this blog for about six months now, and I have found it to be tremendously helpful. The discussions are stimulating, and I love your focus on things that actually have evidence to back them up.

Elizabeth Marsh
Dec 11, 2022 05:23 AM

The question above sounds very much like the program Beyond Textbooks, out of Vail school district(and many others in and out of AZ use it as well). They assess each of the essential standards either every 5 or 10 days, test, then depending on student score, they go through a reteach/enrich process.

Joelle Nappi
Dec 11, 2022 03:05 PM

I appreciate the alternative ideas for monitoring growth, for example in the area of vocabulary acquisition. You mentioned the same kind of monitoring can be done with cohesion and text structure. Can you say more about better ways to monitor those two areas and give an example?

April West
Dec 11, 2022 04:01 PM

+1 in regards to Joelle Nappi's post! I'm trying to wrap my head around a form of reading comprehension assessment that would be truly actionable for teachers.

Judy Mayne Wallis
Dec 11, 2022 05:34 PM

One of the most effective ways to monitor subtle changes in students’ comprehension is through discussion. However, the text matters. Test passages tend to be stilted and poorly written—inconsiderate (Vacca & Vacca). Use of questions in discussion that mirror those on state assessments as formative assessment offers teachers a window into students’ thinking processes and leads to, as Pearson suggests, responsive teaching. Our obsession with weighing the lamb versus feeding makes me wonder if Durkin would find much different results today than she did decades ago. Not much changes without modeling and demonstrating what comprehension looks like and sounds like.

Jen
Dec 11, 2022 05:49 PM

Dr. Shanahan, I found your response interesting.

I am curious what you would say about progress monitoring with Middle School, sixth, seventh, and eighth grade as well as what tools you would recommend.


Thanks for your insight.

Timothy Shanahan
Dec 11, 2022 08:37 PM

Jonathon--

The problem isn't with the tests. They give a reasonable idea as to who is comprehending what they are reading for the most part. The problem is that comprehension is complicated and we are misunderstanding what these tests tell us. They reveal how well students comprehend, they do not tell why -- that would take a much more extensive testing regime that would go far beyond what screeners and monitors are needed to do.

tim

Timothy Shanahan
Dec 11, 2022 08:42 PM

Joelle--

Cohesion can be measured by asking questions that tap into whether a student is following the chain of reasoning -- such as asking who he refers to in the third sentence or checking to see if the student grasps the connection between synonyms -- or checking the meaning of pronouns like any or one... etc. In terms of text structure, with narrative it might be having student summarizing a story on a story grammar chart or with expository it might be asking the student to identify the structure (comparison, problem-solution, enumeration, cause and effect) or, again completing such a chart for a selection.

tim

Timothy Shanahan
Dec 11, 2022 08:45 PM

April--

The point is that since you cannot efficiently and in any meaningful way track student progress in reading comprehension, evaluate how well the student is learning the things that you are teaching. Can they actually apply these skills and abilities in challenging text independently. For example, if you are teaching 6-8 vocabulary words per week, are they learning these words? Are they retaining them over time? Can they comprehend their meanings in text? We know vocabulary growth improves comprehension over time -- so monitoring their success with what you are teaching should tell whether there is likely progress for the student.

tim

Timothy Shanahan
Dec 11, 2022 08:48 PM

Judy--

Actually what the research is telling us is that questions that mirror the skills noted in standards are not giving us a window into children's thinking processes. That's just what they cannot do. They can reveal whether the child understood what he/she read, but not how they came to that answer. -- But your point, engaging students in discussion of what they read does play an important role in children's comprehension development.

tim

Kathy
Dec 12, 2022 02:19 PM

I also need clarification as Tiffany does for younger readers. What are your thoughts on how to progress monitor grades 3-5?

Timothy Shanahan
Dec 12, 2022 02:48 PM

Jen-
What I said for high school would be true for middle school, too. I would likely start the year with a reading test like this:
https://www.serpinstitute.org/reading-assessment
That would allow me to group students appropriately, particularly with regard to Tier 2 interventions. Regular classroom instruction would emphasize fluency, comprehension (strategies, written language), and writing with both narrative and expository text. We'd monitor student progress with what we were teaching.

tim

Heather
Dec 12, 2022 03:15 PM

Thank you for this post. I found it very informative- along with the comments section. Dr. S - thank you for taking so much time to answer all the questions.

Elizabeth Clemens
Dec 13, 2022 05:09 PM

My experience with struggling readers suggests a need for parental concern. The parent should be advised to seek tutoring for this student, who may have missed early reading instruction, for whatever reason. Classroom teachers can't address comprehension problems at this level, mostly because bureaucracy is not addressing the problem earlier. A summer tutorial, offered by private or public, could bring this student's reading level up to grade placement and beyond. This should be the student's and/or parent's choice, once they are informed. Teacher accountability requires truth in education. Don't take the system's problems and make them your own. One on one instruction is needed here.

Timothy Shanahan
Dec 14, 2022 08:45 PM

Kathy and Tiffany--

The comprehension problems that I noted for older students, are also problems in the elementary grades (they are an issue of comprehension not development). However, in the elementary grades it is very reasonable to monitor decoding ability, oral reading fluency, spelling -- and those skills can be profitably examined 3-4 times per year.

tim

J
Jan 30, 2023 10:55 PM

I am wondering about your thoughts on assessing reading comprehension or mastery of a standard solely shown through writing. Each quarter we administer a midterm, final, and 2 formative assessments based on the standards taught during those weeks. We are now adding a weekly writing assessment,based on a prompt that assesses the end of the learning continuum. The students cannt receive support during the first write, but 1-2 weeks later there is a reteach and students would write based on a similar prompt to assess growth. Do you have thoughts on assessing reading comprehension this way?

Timothy Shanahan
Jan 31, 2023 02:19 AM

J-

I like assessing reading comprehension through writing, but it is important to remember that it will under estimate comprehension ability a bit, because performance is affected by both comprehension and writing ability. A student who understands a text may not be able to make it clear that he/she understood the text through writing (but perhaps could orally). This is especially true for second language learners. I'm willing to risk that however because it encourages teachers to engage kids in written response to text -- which is a skill worth developing and that contributes to student learning.

tim

What Are your thoughts?

Leave me a comment and I would like to have a discussion with you!

Comment *
Name*
Email*
Website
Comments

Monitoring the Reading Comprehension of Older Students

24 comments

One of the world’s premier literacy educators.

He studies reading and writing across all ages and abilities. Feel free to contact him.