The Knewton Blog

Our monthly newsletter features edtech and product updates, with a healthy dose of fun Knerd news.

The Real Lesson of PISA

Posted in Ed Tech on June 18, 2013 by

The 2012 PISA results are due out at the end of this year. With their release is sure to come another prolonged period of navel-gazing about what’s wrong with [insert your country here]’s schools, curriculum, teachers, and education system as a whole.

PISA assesses 15-year-olds’ performance in three areas: math, science, and reading. The study started in 2000 and has been performed every three years since.

In 2009, the U.S. landed solidly in the middle of the rankings: 31st in math, 23rd in science, and 17th in reading out of 74 countries. The headlines were dramatic. All Americans left behind! Red scores rising! America’s woeful public schools! U.S. Secretary of Education Arne Duncan proclaimed, “Americans need to wake up to this educational reality – instead of napping at the wheel while emerging competitors prepare their students for economic leadership.”

There are many problems with education both in America and in the rest of world. Dramatically increasing global access to education for those who don’t have it, and quality for all who do, is one of the most important tasks facing the human race today. But what, if anything, do PISA rankings tell us about this?

When I look at PISA rankings, the only problems that are obvious to me are with the rankings themselves and how they are so easily misinterpreted.

Take Shanghai. The city ranked #1 in all three PISA rankings in 2009, prompting much talk on China’s surging dominance. “Wow, I’m kind of stunned, I’m thinking Sputnik,” said former DoE staffer Chester Finn, Jr. But Shanghai was one of few cities on a list of countries. China as a whole has never been represented in PISA or any other international assessment, while Shanghai is a cultural and intellectual magnet for the entire nation. The innate selection bias of it is jaw-droppingly egregious. A fairer comparison would have been Shanghai to, say, Boston – which Boston would have won.

It isn’t just the statistical inferences of PISA scores that are prone to misinterpretation. PISA scores also make it easy to obscure nations’ crucial cultural and policy differences, many of which those nations have little appetite or ability to alter.

Take Finland, another PISA powerhouse. The country has been in the top three of almost every ranking since 2000. Everyone wants to know the Finns’ special sauce.

It turns out their secret is counterintuitive (and not a secret). Finland has pursued a deliberate policy framework that promotes evenness in education. No one attends private school. Rather than focusing on exceptional education for its most gifted students, Finland focuses on overall parity around a high mean. Of course, Finland isn’t doing this to “game” PISA; egalitarianism is central to their culture. But it just so happens that there is probably no better way to top PISA rankings.

Is Finland’s educational policy the right one for every nation? Who knows? What makes a modern society better off, and what does “better off” mean? What if a modern society produces an intensely gifted top decile of students that become scientists, doctors, researchers, and policy-makers, but a relatively weak bottom two deciles that pull down their national average? Mightn’t that nation produce more original breakthrough science and technology? Anyone who truly understands how averages work would never dream of considering a giant gross average like PISA without also taking into account the effects of dispersion from the mean.

America’s educational trouble versus Finland isn’t even ultimately a PISA problem; it is a poverty problem. U.S. students of low socio-economic status are significantly lowering America’s average. If their numbers weren’t included in PISA results, the U.S. would be atop the rankings.[1] It is America’s great socio-economic dispersion from the mean that is hurting its PISA scores versus Finland.

Furthermore, as any businessperson knows, it is relatively easy to affect positive change at small scale. Doing so at massive scale is extraordinarily more difficult – or as one world famous businessman once said, “Well that’s the real trick isn’t it?”[2]

So can large nations learn anything at all from Finland’s example? Probably just one thing: teachers matter. A lot. Despite the great inequality of U.S. society compared to Finland’s, improving the bottom 10% or so of American teachers to be more like 50th percentile teachers – nothing too heroic, just perfectly average teachers – would vault the U.S. past Finland to at-or-near the top of the global rankings.[3] The U.S. could have its cake and eat it too — a deeply unequal society with top PISA scores. That is how influential teachers are to student learning.

The real lesson of PISA, then, is about teacher quality. When the 2012 numbers come out soon, ignore all the hoopla and recriminations. Avoid the media’s clichés about this country versus that, or the growing dominance of China, blah blah blah. There is one thing that PISA says that matters, and only one thing. Supporting teachers as best we can – with more training, technology, and better content – is the closest thing we have to a silver bullet for the education system today. Everything else is just, as Mark Twain would say, “lies, damned lies, and statistics.”

[1] In schools where less than 10% of students receive free or subsidized lunch, the reading score is 551 – just six points lower than Shanghai’s and good for second place. In schools where more than 75% of students receive free or subsidized lunch, the score plummets to 446.

[2] Han Solo