The latest news about US math performance is out, and it's not good. In the most recent international comparison, US students scored 25th out of 30 countries.

But didn't we just read that US students "generally equal or outperform students in other parts of the world"? A related study reported that individual states performed better than most countries.

Last year, weren't we just about average internationally?

And aren't math scores improving?

And aren't reports of poor performance exaggerated?

Well, the story is more complex than the headlines or the quick summaries. The truth is that there isn't any single answer. How we're doing depends on:

• Who "we" are
• Who we're being compared to
• What we're being compared on
• What we consider to be acceptable (or good) performance

Most of the time, when you hear about US math performance, they're either talking about PISA, an international comparison, TIMSS, a different international comparison or NAEP, a US-only study that compares performance over time, nicknamed "The Nation's Report Card." Each of these tests uses different US students, uses different comparison groups and measures different aspects of math performance. In addition, the headlines and commentary put a more or less optimistic spin on things.

PISA

PISA (the source of the most recent headlines) compared 15-year-olds from 30 different countries (in math; they also looked at science and reading). We ranked 25th. I guess the good news is that we ranked above Italy, Greece, Turkey and Mexico (also Portugal, but not significantly). The PISA countries tend to be economically developed nations, so the competition's tough. Still "bottom of the developed nations" isn't exactly a slogan for America's economic competitiveness. We didn't do any better on this PISA exam than on previous ones.

PISA purports to test "mathematical literacy." Some say that isn't really math. Whatever. Check out the questions for yourself. That might not be pure math, but I sure wish our 15-year-olds could estimate the area of Antarctica.

TIMSS

TIMSS stands for Third International Math and Science Study. They liked the acronym so much that they decided not to have a Fourth IMSS. Instead, there's a TIMSS 1995, 1999, 2003, 2007, etc. So, I guess TIMSS doesn't really stand for anything. It's just TIMSS.

TIMSS generally tests students in grade 4, grade 8 and at the end of schooling (grade 12 in the US), though the end-of-schooling test always gets lower participation, and they didn't test it in 2003 or 2007. The countries that participate generally vary from year to year and from grade level to grade level, but there tends to be a wider economic range in the TIMSS countries. Macedonia and Botswana participated in TIMSS, not PISA. On the other hand, Singapore was the best-scoring TIMSS country, but it wasn't in the PISA study.

In TIMSS 2003, the US ranked 15th of 45 countries at the 8th grade level and 12th of 25 at the 4th grade level. This is better than US performance in the 1995 and 1999 variants of the test, and the US beat the international average on the test. So, if you look at TIMSS, we're above average but not in the top tier of countries.

The TIMSS exam questions tend to be good questions that ask students to demonstrate understanding of basic math skills. They don't require the kind of application and integration that makes the PISA questions "math literacy" rather than math. Decide for yourself if that's good or bad.

NAEP

The National Assessment of Educational Progress is just about the US. It is a test given by the Federal government to a sample of US students every few years. Because the test has been given since the 1970s, it gives us the ability to see how well today's students are doing compared to students in the past.

NAEP questions tend to be basic math questions. The general trend is that students are doing better now than they were in the 1970s, especially for younger students, but the increases have not been very dramatic. Absolute performance levels are nothing to be proud of. According to NAEP, the average 17-year is "developing an understanding of number systems." That's not exactly preparation for rocket science. And 17-year-olds have not been improving. Educational policy wonks may argue about whether NAEP scores are improving more under NCLB than they were previously, but these arguments are really irrelevant. The argument is over whether NCLB has increased improvement perceptibly. No one argues that there is evidence of dramatic improvement.

Frankenstudies

Lots of studies try to dig in to NAEP, PISA or TIMSS, often searching for some bits of good news. One example is the study concluding that US students "equal or outperform" those from other countries. This study tried to compare performance for countries on TIMSS to individual states. Since TIMSS wasn't designed to measure performance of US states (just the country as a whole), they used NAEP scores to predict what the states' TIMSS scores would have been, had they taken the test.

There's a lot of clever statistics in there, but the studies end up saying nothing new. In TIMSS, the US did better than most countries, but we're still not in the big leagues. Massachusetts (our top-scoring state) doesn't do as well as Singapore. Despite the optimistic headlines, the studies admit that, on average, US students are not proficient in math.

The Bottom Line

Measuring mathematics performance for a country is complex, and comparing it across countries is even more complex. It is easy to be confused by all the different studies, re-analyses and interpretations.

However, the overall story is pretty consistent. US students are not equal to our expectations in math and science. They aren't meeting NAEP's proficiency standards and they aren't leading the world. At lower grade levels, they're probably above average. At higher grades, they're average, at best.

Younger students in the US have improved, both relative to the past (NAEP) and relative to other countries (TIMSS). Older students have not improved (NAEP and PISA), at least not enough to make any substantive difference.

The Asian countries have shown that dramatic improvements at a national level are possible. We can do better. If we want to lead the world, we need to do better.

PS

No study is perfect, but you shouldn't dismiss any of these results because of fundamental flaws. I've heard people complain that, in the US, everyone goes to school but, in those other countries, only the elites are being tested. Or that we might not do well on average but our best students outscore their best students. Or that we don't score well on tests but we're more creative. The people doing these studies are very, very smart. They've heard all of these objections, and they've tried to address them. Specialists can argue about how well particular issues are addressed, but it is very unlikely that there's some big, gaping hole that invalidates the whole thing.

On the other hand, the PISA guys did misprint the US reading test and had to throw the results out...