Here’s a quick tip: when a study that purports to find evidence of treatment effectiveness–preliminary or not–doesn’t have a control group (a group that doesn’t undergo treatment but is otherwise similar to the group that does), you should raise your eyebrows. Or shake your head. Or roll your eyes. Whichever you prefer.
Why do we need a control group? If the treatment works, we will see improvement in the patients, so isn’t that evidence enough? Well, no.
[T]he whole idea of an experiment is to identify two identical groups of people and then to manipulate something. One group gets an experimental treatment, and one does not. If the group that gets the treatment (e.g., a drug, exposure to a violent video game) behaves differently than the control group that did not get the treatment, we can attribute the difference to the treatment – but only if we can rest assured that the two groups were similar prior to the treatment…
The key issue is that it is always important to have a control group if you want to assess the impact of a treatment.
(By the way, I recommend reading the article I just quoted, “How to Be a Wise Consumer of Psychological Research” from the American Psychological Association.)
In the introduction, the authors of this study make the case that anorexia nervosa (AN) is difficult to treat and difficult to study (low prevalence, high dropout, necessary long-length of follow-up, etc…) and so if we could have treatment approach that would be successful on an outpatient basis, that would be great.
No argument there. There is a lot of consensus that long-term continuous low-intensity care is crucial for sustained recovery. (A short stay at a hospital might get the patient out of the medical danger zone, but it is highly unlikely to cure them.)
CBT, or cognitive-behavioural therapy, is something that could be administered on an outpatient basis, and the authors claim, “is the leading empirically supported treatment for bulimia nervosa (NICE, 2004; Shapiro et al., 2007).” (I’ve got to check those sources myself, before I can feel comfortable claiming that.)
And by the way, I want to be especially clear in posts that I write about treatment: I am interested in evidence. There are a lot of recovery-oriented blogs out there, and plenty of blogs supporting one treatment modality over another. This isn’t one of them. I’m simply interested in evidence, good research and good science (I have no monetary or psychological stake in this.)
Back to the study, the authors wanted to evaluate whether an “enhanced” form of CBT (which “focuses on modifying the mechanisms thought to perpetuate all forms of eating disorder psychopathology”) is effective in treating adult anorexia nervosa patients.
CBT-E is a treatment for patients with eating disorder psychopathology. With patients who are underweight, it has three phases. In the first, the emphasis is on increasing patients’ motivation to change. Then, if willing, patients are helped to regain weight while at the same time they tackle their eating disorder psychopathology including their extreme concerns about shape and weight. In the final phase the emphasis is on helping them develop personalized strategies for identifying and immediately correcting any setbacks.
Patient recruitment was done in the UK and in Italy. All of the patients partook in 40 one-on-one CBT-E sessions that lasted 50 minutes each, over a 40 week period. None of the participants were involved in any other kind of psychological treatment. They were followed up for 60 weeks after treatment CBT-E sessions ended.
Patients fulfilled the DSM-IV criteria for AN (amenorrhea was not a criteria) and were between BMI 15-17.5 (Italy cohort didn’t have a minimum BMI threshold). All were between 18-65 years of age. The majority were single, female, and in their mid-20’s. They had an ED for an average of 3 years. Patients could not have had specialist treatment for AN within the previous year and they could not have a coexisting Axis 1 psychiatric disorder. Thirty-nine (out of ninety-nine) patients were on “clinically warranted” (what does that mean?) antidepressant medication and were kept stable on the medication for the duration of the CBT-E.
So what did they find?
MAIN FINDINGS
- ~ 1/3 of the initial 99 participants did not complete the study
63 patients completed CBT-E, 53 of those were assessed 60-weeks post-treatment:
- mean BMI before treatment was 16.0 (standard deviation, SD 1.2), after treatment: 18.8, and 60-weeks post-treatment: 18.7 (SD 2)
- 39/63 had a BMI > 18.5 right after treatment, and 60-weeks after that number decreased to 30/54.
- before treatment: 27.4%, 20.6% and 7.9% of the sample engaged in binge eating, self-induced vomiting, and laxative misuse, respectively (out of 63 patients)
- 60-weeks post-treatment: 18.5%, 13% and 9.3% of the sample engaged in binge eating, self-induced vomiting and laxative misuse, respectively (out of 53 patients)
- in terms of ED psychopathology (measured by EDE-Q), scores for overall severity decreased across the board following treatment and then increased during the 60-week follow-up assessment, on average of about 85th percentile before treatment to about 50th percentile 60-weeks after treatment (Luce et al., 2008)
Out of the 63 patients that completed treatment, 5 received “significant additional treatment” (the authors did not mention what that entailed) and 7 others received five CBT-E “booster” sessions.
What were the main differences between patients that completed CBT-E from those that did not?
As you might guess, the more severe the eating disorder and general psychopathology, the less likely the individuals were to complete the treatment. Binge eating frequency, purging frequency, and EDE-Q scores were higher in those that did not complete the treatment. Isn’t CBT touted as the most “evidence-based” method for bulimia? It is the “leading empirically supported treatment for bulimia nervosa,” according to the authors. You would think that anorexia nervosa patients that are bingeing and purging would be more likely to reap the benefits of CBT, at least in terms of decreasing the frequency of bingeing and purging, if nothing else. This seems somewhat odd to me.
I mean, I get it, they were sicker overall, but still, I would expect patients that were bingeing and purging the most to see the biggest improvements in that regard–maybe not overall EDE-Q psychopathology but at least for the symptoms that Fairburn claims have the best outcomes with CBT.
Why did patients drop-out of treatment?
Two main reasons: participants either just stopped coming or they were withdrawn because of their physical health or lack of progress.
This is a great way to positively skew the data: you enroll 100 patients, 35 drop because it is really not working for them, and from the remaining 65, half end up improving. When you are counting just the ones that completed the treatment (meaning they were making some progress) about half seemed to benefit. But looking at the whole picture, only a third seemingly experienced benefits from the treatment.
What about maintaining change after treatment?
The third finding is therefore of particular interest. It concerns the stability of the changes obtained. Despite there being little exposure to further treatment, the changes were generally well maintained with there being only a slight deterioration in weight and eating disorder features. This is in marked contrast to the reports of high rates of relapse over the 12 months following hospitalization, even with ongoing therapeutic input (Carter et al., 2009; Kaplan et al., 2009; Walsh et al., 2006).
Of course you’d expect to see that when you only keep the most successful patients, right?
Remember: 99 of patients enrolled, 63 finished CBT-E and 53 completed follow-up. So, in the 53 that completed follow-up there was “minimal” deterioration. But what about the ones that didn’t complete the study? What about the 15% that completed CBT-E but didn’t follow-up? If there was deterioration for notable deterioration in the patients that came for follow-up, you can image the deterioration for those that didn’t could be even worse. Maybe much worse.
After all, if you partook in treatment, and did okay-ish, and then a year later you felt you were back to square one, would you want to go and admit that to the doctors that spent their time treating you and were proud of your original progress? Maybe, but probably not.
And here is another point that’s conveniently glossed over in the discussion: antidepressant medication.
Thirty-nine out of the original ninety-nine patients were on antidepressant medication prior to starting, and following the completion of the study. There was NO mention of the medication status of patients that completed the study. Fifty-three completed follow-up, and as far as I’m concerned, not having this information available, thirty-nine of them could’ve been on antidepressant medication–which they started PRIOR to enrolling in this study.
Here’s a scenario: 39 patients start antidepressants prior to the study, and for some of them, this will decrease their depression and/or anxiety. Say they start this a month or so prior to the study, so when the study starts, they are just beginning to reap the benefits of the medication. Say for those patients that feel less depression and less anxious, they might think less about food or weight, maybe they’ll even start eating a bit more or reduce the time they spend exercising.
And so they begin to get a little better. Antidepressants will not cure anorexia nervosa, but they might decrease anxiety enough to enable some patients to more easily put on a few pounds and maybe even feel more motivated to eat. Of course, this will not be the case for most, or even the majority of those that go on antidepressants, but it will probably be the case for some.
Now given that it is not stated which of these 39 patients completed the study and which dropped (major omission that I think is ridiculous), how can we make any conclusions about the effect of CBT? Out of the original 99, 60-weeks following treatment for the 53 that completed that part of the study, only 30 had a BMI above 18.5, and the average BMI was 18.7.
We simply cannot determine if the positive effects we are seeing here are due to CBT-E, antidepressant medication OR other factors (placebo: regression to the mean, not-treatment related improvement, etc..).
There were more limitations, and although Fairburn et al mentioned them first, I don’t think they were the biggest problems in this study: 1) we don’t know if the findings here can be generalized to those below BMI of 15.0 or above 17.5; 2) we need longer follow-up; 3) this study had only “modest statistical power” for detecting site effects. What does that mean? It means, in short: we need a lot more people if we really want to determine that this treatment works or doesn’t work.
Anyway, I’m not saying that CBT-E is or isn’t effective for the treatment of adult patients with anorexia nervosa, what I’m saying is that this is not something we can determine from this study–at all–because there was no control group. And if you want to to know whether your treatment or intervention had any positive or negative effects, you absolutely have to have a control group.
EDIT: Missie Plotkin rightfully pointed out on the Facebook page that I didn’t emphasize that this was a preliminary study. I mentioned in my first sentence that regardless of whether a study is preliminary or not, controls are important, but forgot the mention this study was meant to be a very preliminary study to provide evidence for larger randomized controlled trials. Nonetheless, any study that is meant to show that a treatment has a positive effect has to have controls for us to draw any conclusions. Otherwise, it is nearly impossible to tell whether the results are due to the treatment or to other factors. So, what’s the point of a prelim study where we can’t tell if the effect is due to the treatment or not, frankly, I’m not sure.
References
Fairburn, C.G., Cooper, Z., Doll, H.A., O’Connor, M.E., Palmer, R.L., & Dalle Grave, R. (2012). Enhanced cognitive behaviour therapy for adults with anorexia nervosa: A UK-Italy study. Behaviour Research and Therapy PMID: 23084515
I am NO fan of Fairburn’s and agree with all you have said here. I would also point out that of the British cohort of 119 referred to the study by their GPs only 50 were able to START the treatment let alone complete it. While for some this will hopefully have meant that they were less unwell than the study demanded of them, believe me, in order to get an eating disorders referral from a UK GP you need to be pretty unwell and I am concerned to know what happened to those 69 people who were ruled out before the study even started.
However I am not condemning the paper entirely. I disagree that it provides “strong preliminary support for this use of CBT-E” but agree that it probably does “justify its further evaluation in randomized controlled trials”
Fiona,
What you said about the referral from a UK GP is really interesting and admittedly not something I considered while reading this study! This makes me think of whether there are different trends/patterns in referrals that GPs make in the UK, versus Canada, US, Australia, etc..
Maybe their BMI was <15.0, which was the minimum for the UK cohort, that seems like a logical explanation based on what you said.
I agree that we need to do more studies simply because I think this study hasn't shown anything! Of course, I'd love to see more studies with control groups (not necessarily RCTs yet, since that's the "gold standard"). I'm all for more research and more evidence-based treatment. I'm just perplexed as to why anyone would do a preliminary study with the purpose of finding evidence for further studies, without a control?? Case studies are fine for qualitative analysis, you know. Let's ask 20 patients about their experiences, and if there's something there that might provide us with ideas on how to develop novel treatment techniques, or how to change the ones we have, or something.
I think I'm just very annoyed because I don't quite see how one can say it is "strong preliminary support" when they haven't actually done an experiment... all of their positive effects that they found could've been due to antidepressant medication, for all we know.
And here's the thing, I don't doubt CBT-E helped some of the patients. I am skeptical that there was complete overlap between those taking antidepressants and those that did well. I wouldn't be surprised if there was NO overlap at all. I wrote that just to make a point. Which is: the authors have to provide this information before they can make the claim that the improvements were due to their treatment. That's all I'd like to see.
My comments:
1. I actually think the study itself is not all the bad – 53/63 is a great follow-up ratio – especially because it is so difficult to keep people in a study for this period of time. . . You’re right that those that dropped out may have been sicker – but they also might have moved (i.e. study took place over 2 years), made major life changes, been unavailable for various reasons – so I don’t think the follow-up issue is a giant problem. In addition, a 1/3 drop-out is pretty typical for a study like this – especially AN which is a treatment group that is notoriously hard to recruit.
2. I’m impressed by their eligibility requirements – especially the part about no comorbid Axis 1 disorders – it would be difficult to find a sample like this:
“Patients fulfilled the DSM-IV criteria for AN (amenorrhea was not a criteria) and were between BMI 15-17.5 (Italy cohort didn’t have a minimum BMI threshold). All were between 18-65 years of age. The majority were single, female, and in their mid-20′s. They had an ED for an average of (dare I say only) 3 years. Patients could not have had specialist treatment for AN within the previous year and they could not have a coexisting Axis 1 psychiatric disorder. Thirty-nine (out of ninety-nine) patients were on “clinically warranted” (what does that mean?) antidepressant medication and were kept stable on the medication for the duration of the CBT-E.”
3. The antidepressant treatment is a problem – and it you’re right to be suspicious about this – In this type of psychiatric research, what is NOT said seems to be just as important as what IS said.
They have no control group but it WOULD have been nice to see an analysis of the anti-depressant subgroup in comparison to the non-medicated group – especially because it WAS 1/3 of their sample, so some statistical analysis WAS feasible. As you stated, it also would have been nice to see them mention the portion of people on antidepressants that finished to trial or were more likely to follow-up.
3. In terms of the results, but you are right in that their analysis is flawed. . . to measure everything against those who completed IS a way of biasing the data.
They should have done this:
http://en.wikipedia.org/wiki/Intention_to_treat_analysis
The Archives Maudsley L&L study used an intent to treat analysis and I’m much more inclined to believe those results that these – especially because I don’t believe Fairburn et al., AREN’T aware of the methodology that is used to best analyze treatment outcomes. . .
A:) thanks for your comment!
1. I don’t think that 84% follow-up from those that completed the study is bad – I think it is good, actually. And you are right, it can be totally not sickness/ED related reasons, like moving. That’s true, I overlooked that. I think my problem wasn’t so much the number that ended up following up, because that’s nothing something they can control to a huge extent (just cross their fingers), it is just that it seems misleading to me to claim that this is “strong prelim” evidence when individuals who are likely (though as you point out, not necessarily) to make the findings seem less promising, are not counted in the finally tally.
I am also not surprised that 1/3 dropped-out. Again, that’s fine. But then I’d expect kind of better results from the ones that didn’t. And also, again, if 1/3 dropout rate is typical for AN, and we don’t really have any evidence-based treatment for adults with AN, then is this really that promising? I mean it could be, certainly, I just don’t think this study showed that.
2. Yes, I was surprised about the Axis 1 disorders but disappointed about the antidepressant treatment. A few others (4 I believe, going from memory) were weaned off before starting treatment.
My main problem again isn’t that they included the antidepressant group. I mean, that’s fine, if you are honest about it. It is hard to find the “perfect” patients who are willing to do a study. That’s not my problem, and if seems that way in the post then I did a poor job of communicating in this post. My problem is that this fact is completely ignored in the statistical analyses and in their discussion. It is easy to just compare the group on antidepressants and one that wasn’t. Just against each other. Even THAT would provide us with some idea about whether or not CBT-E was useful. Was it useful for those on ADs? Not on ADs? No differences? Etc..
So, with your 4th point. They did both, the did do an intent-to-treat analysis, which I left out for space reasons, because I thought the people who completed the study weren’t that convincing either and because I would’ve liked to see in the intent-to-treat analysis, a breakdown of the stats for people who completed and who didn’t. I didn’t like that it was mashed together for those that did and didn’t (as far as I understood), which made it more confusing, so I thought I’d stick to just explaining the data for those that finished. Here is their Results paragraph for that:
“Intent-to-treat findings at end of treatment and 60-week follow-up
Although the primary goal of this study was to determine the proportion of patients with marked anorexia nervosa who can complete this outpatient treatment, and their treatment response, intent-to-treat data are reported in Table 2. The method of data imputation involved moving the last available data point forward as this has been most commonly used approach in the studies to date. It can be seen that there was a marked increase in weight. By the end of treatment the mean BMI had increased from 16.1 (SD 1.2) to 17.9 (SD 1.8) and over the 60-week period of follow-up it remained stable (mean BMI 17.8, SD 2.0). The increase in weight was accompanied by a decrease in eating disorder psychopathology and general psychiatric features.”
I always wish I had more to say because I find your posts very interesting and thought provoking, but cognitively I’m kaput, sorry.
This post reminds me of a local clinic here, that claimed that the majority of it’s patients were ‘cured’ from their eating disorders. The truth was, they very carefully screened potential clients, only accepting those who were ‘dedicated’ to recovery, and, I guess, fit certain criteria. They also dropped you like a hot potato at the first sign of you not progressing to their liking. Of course most of a carefully screened group will have results as you screen them for!
I do think, from the tiny bit of CBT I’ve done so far, that it definitely has it’s place, but ALONGSIDE other therapies/interventions/treatments. Alone, I think it’s nowhere near enough. I’m using it as a tool to cope with trauma therapy. Changing behaviours is a big important part of trying to recover from an ED, but it’s not EVERYthing. Just as ED isn’t just about the eating/exercising/whatever behaviours the person exhibits, that’s just the surface of it really.
Hi Fiona, thanks for your comment and your compliment 🙂
(You are not cognitively kaput – your comments are very thorough and thoughtful and I always appreciate them!)
“They also dropped you like a hot potato at the first sign of you not progressing to their liking.” Yeah, it is unfortunate. At the same time, it is perfectly fine to state for whom the treatment will work best – if it works best for adolescent females who’ve had AN for <3 years, fine, state that.. or for a subset of the original patients, that's fine too. I just think that needs to be more transparent. No one treatment will cure everyone's eating disorder, but I think it is very important to know what will work for whom.
"I do think, from the tiny bit of CBT I’ve done so far, that it definitely has it’s place, but ALONGSIDE other therapies/interventions/treatments. Alone, I think it’s nowhere near enough. I’m using it as a tool to cope with trauma therapy. Changing behaviours is a big important part of trying to recover from an ED, but it’s not EVERYthing. Just as ED isn’t just about the eating/exercising/whatever behaviours the person exhibits, that’s just the surface of it really."
I agree with everything you said, definitely.
“So, what’s the point of a prelim study where we can’t tell if the effect is due to the treatment or not, frankly, I’m not sure.”
The point surely is to do with the inverse scenario, which is likely much more common? I.e. you try lots of stuff, mostly you see no effect, so you move on. If you see an effect, then you do a more rigorous experiment with a control group? This is especially relevant if the setup cost of doing it ‘right’ is a lot higher than a ‘quick and dirty’ study?