Bad Science?

It seems as if every day a new article is published promoting the benefits of meditation. According to the litany of reports, meditation can help with stress, anxiety, depression, sleep, and can even make you a better, more compassionate person. But according to a 2018 meta-analysis of meditation studies, the science behind these claims is plagued by methodological flaws.

The analysis—“The limited prosocial effects of meditation: A systematic review and meta-analysis,” written by Ute Kreplin, Miguel Farias, and Inti A. Brazil and presented in the open-access journal Scientific Reports—looked at studies of the supposed prosocial effects of meditation, such as increasing compassion, connectedness, and empathy, or decreasing aggression and prejudice. They found that “the methodological quality of the studies was generally weak (61%), while one third (33%) was graded as moderate, and none had a grading of strong.” Other investigations, including a 2015 analysis of lovingkindness meditation studies published in the journal Mindfulness and a 2014 lovingkindness study published in the Journal of Consulting and Clinical Psychology, found similar results. And a 2009 meta-analysis published in the Journal of Alternative and Complementary Medicine found that results from studies purporting to show the benefits of mindfulness-based stress reduction techniques also suffered from methodological flaws.

Tricycle spoke to Ute Kreplin, one of the authors of the meta-analysis. Kreplin received her PhD in Affective Neuroscience at Liverpool John Moores University, teaches psychology at Massey University in New Zealand, and has conducted research at Oxford and Coventry universities in the United Kingdom. Here she discusses why meditation studies tend to be flawed and what that means for meditators—if anything.

Do your findings mean that we have to throw out all these meditation studies? It doesn’t mean that we can’t learn anything from them. But we need to be really cautious about how we interpret the findings that come from them and to keep in mind that they do have limitations. We could use them as a jumping-off point to create good studies, and then we can get good scientific evidence rather than stuff that’s biased.

Why is there so much biased research on meditation? There are many parts to that. One of them is that we normally study what we feel passionate about—and that’s not a bad thing. We want to do something that interests us, and we want to go and find out about it. But then it’s easy to get carried away wanting to prove that something works, and we tend to ignore some of the things that don’t work.

Another factor is that for a very long time it was important that you publish positive results. So if you ran a study and you didn’t find anything, then you had wasted a year of your life [laughs] because no one wanted to read about it. That’s starting to change now. Studies that have negative results or don’t find anything but can show that they are methodologically sound get published anyway. But for a long time there was this pressure, because if you got published, you’d get the money.

You said that people tend to study what they’re passionate about. What brought you to do this study? Are you a meditation enthusiast or practitioner? Not really. I’ve dabbled in mindfulness from my time working as a clinician, but my main research area is actually in neuroscience. It was through coincidence that I got to work with Miguel Farias, who is my coauthor.

I didn’t realize that the findings would be what they were. I thought we’d have a look and see whether or not the evidence showed that meditation increases prosociality. I wasn’t expecting to find all these weaknesses in the methodology. It’s hard to have a perfect study—no study is really perfect—but the number of flaws was quite surprising.

Could you explain what a strong study would look like? A strong study would try to eliminate bias. In drug trials, for example, you would run a placebo condition. Neither the researcher nor the participants would know who gets the real drug. It’s harder to have a placebo in meditation studies or behavioral studies in general, but there are other things you can do.

For example, you can blind your data analysis, so the person looking at the results doesn’t know which group is the control, which in this case would be the group that does not receive the meditation instruction. That way you can prevent yourself from tweaking your numbers to get one group to look better.

Another thing that you can do is preregister the study. There are websites where, before you run the study, you can lay out your groups, the criteria by which you exclude certain participants, and how you’re going to analyze the data. That means that later you can’t change the way you do your analysis to get some extra results. In a preregistered study, additional tests aren’t totally forbidden, but you have to justify them, and “my first one didn’t give me the right results” isn’t a very good justification.

A third thing you should always do in a behavioral study is to have an active control group. Many studies compared mindfulness to doing nothing, which is a passive control. And obviously, if I sit around for eight weeks and think about how I can be more compassionate toward other people, then at least temporarily I will be a little bit more compassionate compared to just going about my daily life. An active control group does something similar to the intervention that you want to test against. The control group could be a discussion about compassion. Then you would have more clarity as to whether the mindfulness practice was more effective than just thinking about com-passion—which would indicate that the mindfulness practice itself led to those results.

Did any of the studies that you looked at use an active control? There were some that used an active control. And some used both active and passive, which is great. But quite a lot of studies just used the passive control group.

Your analysis found that many of the meditation techniques in these studies were taught by one of the authors, which led to bias. Normally in a double-blind study the tester doesn’t know what they are testing for. But in Buddhist traditions, there’s a long process of becoming empowered to teach meditation. So is it possible to have an unbiased person running the meditation study? In the ’70s someone designed a double-blind study of transcendental meditation in which he wrote a whole new script for the control group that did similar things to transcendental meditation but left out some of its components. That’s very time-consuming and very hard. And whoever would be teaching it would have to be naive enough not to realize that one of the meditation scripts is just a sham.

“It’s hard to have a perfect study—no study is really perfect—but the number of flaws was quite surprising.”

Perhaps for meditation studies it’s more important to blind the data analysis and to have an external teacher run the intervention. The teacher can be really experienced in meditation techniques, but he or she wouldn’t know anything about the rest of the study.

One of the findings that surprised me the most was how many researchers were meditation practitioners and also ran the whole study—and yet they still weren’t trying to minimize any of the experimental bias. Experimental bias is quite a longstanding concept in psychology—you learn about it as part of your undergraduate degree. It’s really easy to influence people inadvertently.

The main thing your study looked at was findings of prosociality. How exactly do you define prosociality? Most of the studies we analyzed looked at whether or not meditation or mindfulness increased compassion. Or if people felt more empathic afterward. Or if people were less aggressive or less prejudiced toward minority groups.

How do you measure compassion and empathy? There are two ways. You could do a survey, which is what most people do. But all surveys rely on people to self-report. There were also a few studies that tried to use behavioral tests. So for compassion, for example, there is a classic test. You stage a waiting room with three chairs. Two stooges are hired by the researchers to sit, and then a study participant sits in the third seat. Then a fourth person walks into the room, and then the question is: will the participant give up the seat for the fourth person? The more likely people are to give up the seat, the more compassionate they are.

That sounds like it’s also more costly and time-consuming. A little bit, yeah. Because you need people that help you with the research in order to do that.

Did your study support any claims about the benefits of meditation? I would be cautious about saying that we supported any benefits. The one that came out clearest was an increase in compassion. But most of the findings on compassion fell apart when we looked at these methodological flaws. Compassion was supported only when the people used the passive control group—when it was compared to doing nothing—but it wasn’t really supported when it was compared to active control groups. Also, compassion was supported only when the researcher was part of the intervention team; when we separated out the studies where the researcher didn’t participate, those results fell apart again. We just don’t know. We don’t have evidence against it, but we most certainly don’t have evidence for it, either. What we are trying to say is that we need more good research to get a clear picture of what’s actually going on.

So basically all the articles in all the health magazines touting the benefits of meditation are groundless? Basing their arguments on weak evidence, I would say.

Related: Don’t Believe the Hype

That’s a kinder way to put it. I’ve read other meta-analyses to see if this was only an issue in studies that looked at prosociality. And most meta-analyses or reviews found methodological flaws around the meditation studies. Some areas will really have better evidence than other areas. For example, in cognitive behavioral therapy, the mindfulness-based stress reduction programs are more researched, although I don’t think all the studies in that field are totally sound. We have to be careful when we generalize.

There are many who would argue that Buddhist practice has to be a holistic approach. It involves studying the dharma—the teachings—and a variety of rites and practices, of which meditation is just one. Maybe this question is outside the scope of what you’re doing, but is it possible for randomized scientific studies to explore a holistic method by breaking it down into its component parts? You can compare a long-term meditator with people who don’t meditate and then see how they differ in either their behavior or how their brain activation differs from other people. Then you can say that their lifestyle, not just meditation, brings about these changes. So that question can be investigated with the scientific method to a certain extent.

The problem with those studies is that you can’t determine what the effect might have been attributed to. If someone is a Buddhist for ten years, meditation isn’t the only change in their life. They also might follow a specific diet or they might follow a certain rhythm throughout their day, and all of these factors can have an impact on how you behave afterward.

We looked at studies called randomized control trials, which all had a measure before and a measure after the intervention. The participants were randomly selected to be in the control group or the intervention group. These types of studies are the best way to determine if there’s an effect caused by a particular practice.

In these studies, people made a very specific claim that meditation removed from everything else can have these great benefits—and these benefits can be seen in short-term interventions, not just long-term meditation. It’s a different question behind that: can meditation, when removed from its framework, have the same effects or not? That’s probably more a philosophical question than a scientific question.

Related: Living by Meditation Alone

Are you familiar with the monk Matthieu Ricard, who after being put in an MRI machine was given the moniker “happiest man in the world” (a title I don’t think he’s fond of)? Is that the purpose of Buddhism, though, to be happy?

That’s a good question. Probably not. It’s probably beyond that. Meditation or mindfulness by itself is often sold as the happy pill. The purpose becomes completely different from what it would be in a spiritual context, and you are meditating just to be happy. But is meditation really the right path to happiness? According to research from Dr. Willoughby Britton of Brown University about meditation’s occasional negative effects, as well as earlier accounts of meditation not leading to desired results (such as a 1976 study by Arnold Lazarus of Rutgers University), the answer is: not always.