An Interview with Neal McCluskey: Are the Scores the Same, Significantly Better, Regressing to the Mean, or Slumping Overall?

Dec 15, 2012 by

Neal McCluskey Cato Institute

Neal McCluskey Cato Institute

Michael F. Shaughnessy –

1) Neal recently, there was a Cato@Liberty post on the latest international achievement outcomes of interest.

Can you give us a brief summary?

Sure thing. On Tuesday scores from the 2011 administrations of the TIMSS science and math tests, and PIRLS literacy test, came out. Those are international exams taken by students in roughly 50 to 60 countries as well as a few U.S. states and other sub-national jurisdictions. For the United States what they revealed were pretty high scores relative to other countries save those in east Asia — which always seem to dominate — and small-to-appreciable increases in scores over time depending on the subject and grade level. All-in-all, not a bad showing for the United States.

2) Neal, U.S. Scores Seem to be Up, but Why?<> What are your thoughts?

My main thought is that the scores don’t in any definitive way tell us why they moved as they did. Some people will want to give No Child Left Behind credit for improvements, for instance, but 8th grade math scores rose much faster before NCLB than after. My preference, from a policy perspective, would be to credit school choice, but while U.S. scores have risen along with increasing choice, scores in Scandinavian countries seemed to go down though they have increasingly embraced choice. Meanwhile, research suggests that east Asian nations outperform the rest of the world because of a shared culture that highly prizes academic achievement, especially on material that lends itself easily to testing. But these scores don’t provide enough information to substantiate that, either.

3) There seems to be ” bounteous new international academic achievement data”… “from the TIMSS and PIRLS battery of tests<>.  Is there good news, bad news or no news?

Generally speaking, good news. But it’s also worth noting that all the participating countries have basically the same system: a government monopoly with slightly varying degrees of marginal freedom allowed. So doing well versus other countries likely means doing very poorly compared to what a true free market in education would produce. It’s basically  comparing two- and four-door Trabants.

4) How are the East Asian nations doing ?

Very well, at least on test scores. Interestingly, though, many of those same countries are trying hard to deemphasize testing and get better at fuzzy things like “critical thinking” that they believe we are good at. It is very important to understand that testing does not necessarily capture all — or even most — of what people want and need out of education.

5) Let’s talk math—How have U.S. scores changed?

On 4th grade mathematics we’ve seen pretty sizable increases over the years, going from 518 (out of 1000) in 1995 and 2003, to 541 in 2011. 8th grade scores were also up, but only from 492 in 1995 to 509 in 2011. And scores rose only seven points between 1999 and 2011.

6) What about science?

4th grade performance was essentially static: 544 in 2011, versus 542 in 1995, with a dip in the line in 2003 and 2007. 8th grade also saw some interesting kinks: The high score was 527 in 2003, but 2011′s score of 525 did beat 1995′s 513 showing.

7) Finally, how about literacy?

Only 4th graders are tested in PIRLS, and data only go back to 2001. Again there was a dip in the middle, but in 2011 the U.S. average was 556, versus 542 in 2001.

7) What age or grade are the pupils?

It depends on the country, but the ages are roughly 9.5 years old and 13.5 years old. The grades are roughly 4th and 8th, but different countries sometimes use different grades.

8) Neal, help me out here- what does PIRLS stand for?  Now what exactly is this TIMSS?

This has changed a bit over the years, and I’m not sure how readily the tests are identified by anything other than their acronym. That said, last I checked TIMSS is the Trends in International Mathematics and Science Study, and PIRLS is the Programme in International Reading Literacy Study.

9) What is the real important issue or question here?

The crucial question is why we see the score movements in the United States and abroad that we are seeing. The test scores alone tell us little of use for answering that question.

10) What about confidence intervals and regression to the mean- are they relevant here ?

Standard errors are reported, and certainly there is some play in whether the average for test takers is true for entire populations. As for regression to the mean, the goal of the tests is to measure achievement on set scales, and it seems reasonable to think that all countries could make continual progress by improving numerous things that affect scores. I don’t see evidence — at least quickly looking over the results — of overwhelming regression to the mean either within or between countries. It could explain some changes, but I don’t see anything right off the bat.

Print Friendly, PDF & Email

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.