7 Useful Tricks for Separating Fact from Fiction in ‘Breakthrough’ Health Studies

Not all health statistics are created equal. Apply these tips so that you can be the judge.
Avatar:
Author:
Publish date:
Social count:
40
Not all health statistics are created equal. Apply these tips so that you can be the judge.

Let’s be honest. Whose ears haven’t perked up at the latest study determining what may cause or prevent cancer, rid children of peanut allergies, or prevent miscarriage? The way news outlets report on health, you’d think there’s a breakthrough every other day.

For better or worse, we live in an age of unprecedented technology and the copious amount of information that comes with it. Those of us without medical or advanced science degrees (hand raised!) can find it difficult to cut through the drama to understand what the health statistics we read really mean.

These tips from the experts can help you separate fact from hyperbolic fiction, so you can be your own best advocate without donning a lab coat.

01. Watch the language.

Terms such as “breakthrough,” “medical miracle,” and “game changer” are often markers of inflating the truth. A headline’s job is not to tell the whole story, but to get you to continue reading. The media can harbor an interest in manipulating scientific research to make studies sound more compelling than they really are (can you say “click-bait”?).

Proponents of the “slow science” movement argue that good science requires time for rumination and false starts, while the culture of contemporary science—bolstered by increasing competition for grant money and press—has come to value the quantity of studies over quality.

02. Go directly to the source.

Contributing correspondent for Science John Bohannan warns of “treasure hunting” in multidimensional studies: scientists or reporters may pick and choose among findings to make them appear more significant than they really are.

For the full picture, head to the study itself. Websites like PubMed and Health News Review provide abstracts and objective commentary on medical studies. Look for information about how many people (not rats or guinea pigs) were involved in the study, over what period of time it was done, what data was measured and under what conditions. The more people, the longer and more randomized the study, the better chance the findings can be deemed reliable.

Be wary of percentages: absolute differences, calculated by subtraction, can paint a different picture than relative differences, calculated by ratio. For example, if an allergy resolved in four of a hundred patients in a control group and six of a hundred patients in a treated group, the absolute difference would be two of a hundred patients. You could also reasonably say that 50 percent more patients saw the allergy resolved when treated. Same story, different angle.

A sound study will provide a measured perspective on the results. Do the positive outcomes strike a realistic balance with the trade-offs, be they time, money, side effects, or availability of the drug or treatment? If it’s all good news, something’s probably missing and you should conduct further research by reading other articles or speaking with health or medical experts on the subject.

03. Speaking of sources, there ought to be more than one.

Be cautious of single-source stories. The “research” that goes into these is usually a press release sent out to editors and publications, rather than true reporting. Brooke Gladstone, host of On the Media, cautions, “The pay-to-publish science journal industry is notoriously easy to buy into.” In these publications, there’s an inherent conflict of interest and a potentially harmful lack of true journalism.

Remember that scientific evidence comes by way of data, not testimonies. At Verily, we link directly to studies in peer-reviewed journals so we know they've survived the scrutiny of many other experts. Overall, peer-reviewed scientific and scholarly journals are a more reliable source of information because they are more likely to be unbiased and objective. OMICS provides a list of peer-reviewed open access journals and EurekAlert provides a list of both open access and subscription-based journals. Even then, pay attention to who funded the study for any professional or financial associations that may be a conflict of interest.

04. Understand the differences among clinical trials.

Often a report will mention the stage of clinical trials in which the startling finding came to light. What’s rarely, if ever, mentioned is what those stages are.

According to the U.S. National Library of Medicine, “[c]linical trials are conducted in a series of steps, called phases—each phase is designed to answer a separate research question.” The first two phases look to determine safety, doseage, and side effects in increasing sample sizes. The third phase delves deeper, comparing results from the drug or treatment to current solutions. It is only the fourth phase that looks at various populations and effects of long-term use.

A reference to a phase 1 or phase 2 clinical trial may be exciting scientific progress, but it’s not enough to consider the problem solved.

05. There’s a big difference between causation and correlation.

In seeking a digestible conclusion from medical jargon, it can be tempting to mistake correlation for causation.

Take the “rules” of pregnancy. The lists of what expectant mothers should and shouldn’t do is extensive and can be exasperating: no sushi, no sliced lunch meats, no tuna, no soft cheeses. For crying out loud, what’s a hungry pregnant woman to eat for lunch?

When economist Emily Oster was expecting her first child, she applied her professional skills to assess a number of commonly accepted notions about pregnant women’s “do's and don’ts,” among them the idea that expectant mothers should not drink coffee. After she combed through the available studies—surprise, surprise—her own findings didn’t point to the popular wisdom.

The common claim, she says, invokes causal reasoning: women who drink less coffee are less likely to have miscarriages. But the data actually indicates a correlation: women who consume coffee have higher rates of miscarriage. In her findings, women who drink more coffee also tend to be older, so we can’t be sure which variable—coffee or age—contributes to the rate of miscarriage. Even die-hard coffee drinkers sometimes find the stuff undrinkable in the first trimester, rendering a sound study difficult to perform.

The most reliable study Oster identified showed that reducing the rate of coffee consumption during pregnancy yielded the most promising results for a healthy outcome. In the end, the amount of coffee consumed wasn’t important, only the reduction. Myth busted, lesson learned.

06. Not every condition is a disease.

If it seems there’s a new condition requiring medication every time you turn on the TV, you’re on to something. Some researchers suggest a phenomenon they’re calling “disease mongering,” or “widening the boundaries of treatable illness in order to expand markets for those who sell and deliver treatments.” Ordinary aspects of aging like menopause and balding, as well as risks like high cholesterol and osteoporosis, are posed as diseases. The public’s induced fear can then be capitalized upon to sell new products.

Consider the oft-debated Gardasil, which is recommended for children ages 11 or 12. Human papillomavirus (HPV), the primary condition that Gardasil is supposed to prevent, is a sexually transmitted infection. The drug’s “Who Should Get Vaccinated” page suggests, “The time to vaccinate is before there is any exposure to these HPV types,” without explaining how exposure occurs. Call me old-fashioned, but do we really expect the majority of 11- and 12-year-olds to be sexually active?

This is only one example of pharmaceutical marketing that is not telling the whole story. Peer-reviewed research site PLOS Medicine suggests such efforts are “more often designed to sell drugs than to illuminate or to inform or educate about the prevention of illness or the maintenance of health.”

Consider the source, as the old adage says. Is it someone or something you can trust?

07. Trust your gut.

The one thing you can always trust is your instinct. Of course science isn’t all bad, but it also doesn’t happen overnight. When something seems to be too good to be true, it probably is. The best prescription may be a healthy sense of skepticism and proactive investigation in the face of sudden medical revelations and discoveries.

If something seems relevant to your health and wellness, dig in, read the whole article (don’t skim!), check out the original study or similar studies mentioned, and then do some research of your own (ever heard of Google Scholar?). You owe it to your body and your mind.

Photo Credit: Zitona