���ϳԹ���Դ

While mature evidence has survived years of scrutiny, a surprising preliminary result could easily be overturned tomorrow. Yet, reporters, advocates, marketers and even researchers do not always distinguish between them.

Politicians, educators and public health leaders build policy on a promising early result, call it “evidence-based,” and then watch the policy fail to deliver as the science evolves.

One study is not enough

— PERLS (Policy Evidence Readiness Levels) — offers policymakers a structured way to classify evidence maturity and assess what “follow the science” actually supports at each level.

It also offers an intuitive rule of thumb: When deciding how much confidence to have in a claim, ask whether it is backed by a single study, many studies or something in between.

When a claim rests on a single study, or even a handful of studies, treat the results as possible, not proven. The finding might hold up under scrutiny. But it might not.

There are many reasons preliminary results fail to hold in the long run. Some fail simply by chance. Run enough studies and a few will find striking results purely by luck, the way a coin can land heads five times in a row without being a two-headed coin. Keep flipping, and the truth eventually reveals itself.

Other results fail for less innocent reasons, such as poor study designs, mistakes or cherry-picked reporting of results.

What happens at scale?

Even results from carefully conducted studies often fail to hold once scaled.

For example, studies may show that shifting to a among the closely watched, motivated teams that chose to participate in a study and try such practices. But will the results hold if the practice expands across a broader set of employees and organizations?

Other effects only reveal their real impact over the long run, such as a diet that produces early weight loss but is impossible to sustain, or leads to side effects in the longer term.

The case of phonics

Confidence comes not from a single study but from an accumulation of evidence. Do the same results show up again and again across labs and research teams, and in real-world settings with different types of people? We should have more confidence in a finding that survives many studies than a surprising one that first makes headlines.

This is the scientific process, , where standards of care require evidence to accumulate into , and ultimately guidelines from respected bodies of experts.

When we skip this process, problems emerge. Many educators used a “studies show” argument when replacing phonics with whole-language reading instruction in the 1980s and 1990s.

Decades later, it is clear now that the . Schools are now shifting back to embrace phonics and the science of reading, backed by .

Previously, educational policymakers failed to question the maturity of the evidence that was used to justify such a major decision.

Act now, remain flexible

We often have to act before evidence is mature. There is a health scare, a market shift or a kid struggling right now. Acting on the evidence we have is not a mistake.

The mistake comes from making commitments as if preliminary insights cannot change.

So, when should we trust the evidence, and the experts explaining it? This is not an all-or-nothing question. We should ask whether the claims are based on a single study or an accumulated body of research established across many studies. We should vary our trust and actions accordingly.

, Professor, Jarislowsky-Deutsch Chair in Economic & Financial Policy, and Director of the John Deutsch Institute for the Study of Economic Policy, Department of Economics,

This article is republished from under a Creative Commons license. Read the .