Safeguarding the Integrity of Clinical Research
An hour with Joel S. Perlmutter, M.D.
Dr. Perlmutter has frequently made scientific news for his work in movement disorder research. Recently, however, he made national news for the bold stand he took against the FDA when they approved aducanumab (Adulhelm) for Alzheimer’s disease. Dr. Perlmutter talked with the HD Insights editors about that decision, the environment that may encourage biased decisions and shortcuts to approval, and the illogical leaps that too often characterize how research is interpreted today. He also offers advice to the HD community on how to navigate this environment, protect research integrity, and protect patients from thinly tested or prematurely approved therapies.
One of the things our readers love about HD Insights is just learning about interesting people in the Huntington’s disease community, and you are certainly one of them. I think a lot of us really applauded your integrity with the decision to leave the FDA advisory committee. Would you tell us why you resigned?
Yes! It was a 15 minute decision. I was serving on an 11-person Scientific Advisory Committee to the FDA. Our purpose was to advise the FDA on whether the research evidence on Biogen’s aducanumab for Alzheimer’s disease was sufficient to approve the drug. We looked at the data and there was no evidence of clinical benefit. Ten of us said “no,” and one person was “uncertain.” Several months later, we came to find out that the FDA approved the drug, based on a subset of patients showing reductions in amyloid plaques in the brain, measured with PET scanning. I quit the committee on the spot, not because they didn’t take our advice (that should not be a requirement), but because they clearly and rather astoundingly made the wrong decision from the standpoint of research interpretation.
There were four serious problems with it: 1) the approval was based on post-hoc analysis of a surrogate endpoint in a subset of patients, 2) PET measurement of amyloid plaque in the brain does not have a proven causal relationship with AD signs and symptoms, 3) according to the FDA’s own biostatistician, the change in this PET measurement did not relate to any clinical benefit in this study, and 4) the drug’s side effects were frequent and potentially serious.
Can you elaborate on each of these problems?
Sure, let’s take the first. Amyloid plaque volume was not the primary endpoint of the study. The primary endpoint was clinical benefit in terms of reduced or deferred signs and symptoms of Alzheimer’s, and taking all of their data together, there was no significant evidence of clinical benefit.
Further, amyloid plaque, measured with PET, was only done in a subset of patients, and this did not relate to clinical benefit in these patients. Biogen and the FDA tried to argue that the drug had modest benefit if they only looked at some of the participants in this study. That is called a retrospective review of the data rather than the planned analysis. If you are doing post-hoc cherry-picking, you can use those findings as the basis for a new clinical trial to test the drug in a new group of patients. But to draw conclusions without that new prospective study is dangerous.
It’s like when I drive down the highway here in Missouri, I see all these bullet holes in the center of these targets on barns or stop signs, and that is because people shot the bullets and went back and drew the targets. That is what a retrospective analysis does. We have to guard against drawing targets after the data are collected. That does not provide proof of effectiveness.
On the second and third points, it is easy to jump to the conclusion that reducing amyloid plaque means you are affecting the disease. The rationale for their approach in this case is based on the very rare types of Alzheimer’s disease caused by known genetic mutations that directly lead to build up of amyloid in the brain. Although people with sporadic Alzheimer disease (the 99% of people with AD who do not have a known genetic mutation) have such amyloid plaque in their brains, it remains uncertain whether those plaques actually cause brain damage.
Alternatively, amyloid plaque could be a protective response to the disease pathology. Some investigators believe that Lewy’s bodies may have the same role in Parkinson’s. We just don’t know. Similarly, in Huntington’s disease, it is tempting to assume that lowering the biomarker mHtt as measured in the spinal fluid would be curative, but studies have yet to bear this out.
My concern has to do with the use of biomarkers — which are markers of some underlying disease process. While some biomarkers can reflect brain dysfunction, they may not reflect the cause of that dysfunction. The way I tried to explain it to a congressional committee was, if you go into a cemetery and see headstones, those are biomarkers of people who are dead. If you then remove the headstones, you remove the biomarker of the dead people, but they do not come alive again. That’s the problem. Amyloid plaque may just be a biomarker of where something bad has occurred.
Again, in Huntington’s, we don’t know that lowering mHtt or reducing the brain aggregates from the abnormal huntingtin protein has a clinical benefit. There was a HD mouse model study done years ago showing that if you reduced the aggregates, you enhance rather than reduce toxicity. So administering a drug that reduces aggregates could, in fact, be a dangerous approach. We don’t know, and so conditional approval and emergency authorization of drugs that affect biomarkers without evidence of clinical benefit should be avoided, or at least approached with great care.
The fourth point speaks for itself. When you are dealing with desperate families faced with a devastating disease, their willingness to take risks is high, and as physicians, it’s our job to educate and protect them from risks of severe side effects, especially when there is no proven upside.
In summary, this whole situation is an example of bad interpretation of a pharmaceutical company clinical study. The initial study design was quite good, as pharma studies are in general. The problem is not the quality of the studies, but rather interpretation of the results. The bottom line is that clinical benefit must be proven and that requires well-done prospective studies, not just retrospective analyses of a select group of participants in a study. If a study fails its prospectively planned analysis and a retrospective review suggests that a certain group may benefit then one must do a new prospective study. It’s perfectly legitimate to do that new study to prove effectiveness. Otherwise, may make a seriously wrong decision.
What is going on at the FDA that encourages or allows such decisions to be made?
The FDA right now has several problems. One is the possible effects of conflict of interest. The FDA budget comes from different sources — largely the congressional budget and applicant fees. The FDA has different groups that deal with different parts of their mission, and these groups have very different funding models. It’s crazy — if you look at the part of the FDA that regulates agricultural products, 99% of the budget comes from Congress. But if you look at part of the FDA that regulates human drugs, it’s upside down — 65% funded comes from industry applicants. That’s a serious problem.
The other place where conflict of interest may come into play is fluidity between employees and applicant companies. Somebody works for the FDA, then after leaving the FDA they may obtain a job with a relevant company. This could potentially bias decisions.
Another factor that may influence the FDA is the huge pressure from lay organizations. In the aducanumab case, the Alzheimer’s Association of America pushed hard for approval. I am not aware if there is a relationship between Biogen and them, but I do know the association’s website promoted this drug with a posting that was almost word-for-word the publicity notice that came out from Biogen. I can appreciate that the association represents family and patients with a terrible, incurable disease. But our job is to protect vulnerable populations, not placate them with platitudes and stuff that may not help them. Rather, we have to take the time to educate them.
There was a really nice study addressing questions about how the FDA functions, stimulated in large part by what happened with aducanumab. It was done by a couple of people at Stanford working with a couple of previous FDA staff members. They asked the question, “What happens to applications to the FDA that are initially rejected for lack of efficacy?” They looked at records over five years, and during that time, there were 917 applications, with 123 that were rejected. About 23 were rejected for lack of efficacy, and of those, a fair number were just redirected to a different FDA committee and approved without change. Another group of these 23 were approved without change by the FDA director of the very committee that had just rejected the application. Only in four cases did the applicant organizations provide new pivotal key data that then led to approval. Ideally, that is what we would want with all of these initially rejected applications.
The article also pointed out that it was very challenging to find this kind of data — there was no curated database in which to search for the FDA’s various committee deliberations. Finally, the third thing they emphasized was that in no instance did one of these FDA decisions on rejection refer to past decisions by the FDA. So there was no evidentiary information like you have in the judiciary system to establish patterns and precedents.
Congress could fix the conflict of interest issue. They could fund the FDA 100%. Talk about return on investment! Imagine the money the federal government would save on reduced waste in Medicare expenditures alone. That would more than cover this difference in the budget.
Can you offer some opinions on the state of research design in contemporary clinical trials?
We have good data that study design by either pharma or academics is equally good. The problem is with reporting. Pharma-sponsored studies in general tend to overgeneralize and tend to underreport side effects. And there is plenty of data about that.
In the Biogen case, they should have redesigned a new prospective study based on their retrospective analysis, and done this before receiving FDA approval. But they were anticipating
$6 billion in first-year revenues for a drug that would be given to each patient every for the rest of their lives at the initially established drug cost of about $56,000 a year per patient. And that is just the drug cost. It doesn’t include any MRI scans, the infusion center bills, and potentially PET scans. All of these things stand to probably double the cost to the healthcare system—to us.
Can you shed some light on the history of transparency and the FDA?
I was only on the Scientific Advisory Committee for six years, and when the FDA overruled our committee recommendations, that was the first time I had been reviewing a case. It had happened once before on my committee, but I had not been able to participate in that particular review. Interestingly enough, that wasn’t as lopsided a decision as this. And amazingly, in this case, nobody spoke up to really nail this. Nothing happened. So the aftermath of this aducanumab decision is really opening up a window for us to peer in and look at what’s going on behind the scenes.
Separately, I will tell you that back in 2010 I had a very strange experience when I gave an invited expert witness lecture to an FDA advisor meeting on neuroimaging in Parkinson’s. I was not told what the committee was going to evaluate. The gist of my lecture was that there was no clinical utility for using molecular imaging to take care of people with Parkinson’s. It’s great for research, but has no role in clinical care.
After I finished my talk I was asked to sit in the audience, but told that I could not ask or answer any questions as the applying company gave a series of lectures. Apparently, rather than applying for approval of an imaging agent to be used as a diagnostic tool, the company changed the application to measure dopamine transporter (DAT) in the brain. They gave a series of talks, and when the FDA committee began their open deliberations, one of the committee members wanted to ask me a question. That person was told by the head of the FDA committee that they could not ask me anything. At that point, I wrote an article, “FDA Inquiry: Who wants to know the truth?” about that event.
The bottom line is that this approach dramatically changed in the years after that, which is why I agreed to join the Scientific Advisory Committee. The new norm was that everybody was told what the application was for, it was publicly announced, and all the deliberations of the committee were entirely open. The company would give a presentation, the FDA would give their independent interpretation of the data, then the committee could ask questions. The public could also ask questions, and this was all streamed online, with transcripts and everything openly available. The committee members would vote and then we would go around the room, say how we voted, and provide our rationale. Our decisions were done in the light of day. I was really pleased with how this went while on the committee.
That’s why this situation with the FDA and Biogen was such a disappointment. It was unprecedented in my experience. Even after the head FDA biostatistician said that the changes in amyloid imaging did not correspond to clinical benefit in any way at all, the FDA subsequently approved it on the basis of the amyloid imaging.
Despite questionable clinical benefit and questions on whether changes in this biomarker related in any way to clinical benefit, there were the well-documented side effects of 20% of participants having microbleeds and 35% having focal edema in the brain! The chances of serious side effects could increase once this is applied throughout the country, as some people may take drugs like blood thinners, aspirin or even ibuprofen that could increase the risk of major brain bleeding… and for no clear clinical benefit! So that’s why I stood up and resigned. I did not want to be a part of that decision.
The recent trial with tominersen brought up some intresting issues with the concept of post hoc analyses and their role in clinical trials. Can you talk about this, given your experience?
This is an easy answer. I’m going to quote something the same FDA head who approved aducanumab once said to me: “A retrospective analysis is a reasonable way to generate a hypothesis and test it in a prospective study….but it rarely proves to be true.”
When is accelerated authorization or conditional approval of a therapy appropriate with a devastating disease like HD?
Accelerated authorization was used for aducanumab. This authorization is appropriate when we have a rapidly progressive, untreatable condition that can have serious consequences, for which we have no current treatments. If we have reasonable evidence for benefit of a therapy with acceptable side effects, accelerated authorization can save lives. Emergency authorization is related, and this was applied to COVID-19 vaccinations. This is a situation in which the FDA acted brilliantly in my opinion and made a decision that has been important for the healthcare of this country and the world, quite frankly. So these approaches can play a very important role.
Conditional approval is a different issue. This indicates that the FDA gives approval if the company follows up with another study. In the case of aducanumab, the FDA initially approved it, which was not justified, in my opinion, for all people with Alzheimer’s, yet the study specifically focused on early AD and Biogen made a big deal about that fact. And the FDA was requiring Biogen to do another prospective study to prove clinical benefit that would be conducted over nine years. In my mind, this is billions of dollars down the road and could bankrupt Medicare. In this case, I believe that the conditional approval was too late and the study should have been done prior to approval.
The FDA has made it very clear that they don’t engage in cost discussions. They just approve the drug, yes or no, and they defer the cost of drug discussions to CMS or insurers. Do you think that bright line needs to be erased and that decisions on cost need to be engaged in during the approval process, or does that line function well?
I’m not sure I have the answer on this one, but I think I would personally prefer the FDA to stick with risk/benefit assessments. That is their expertise. If they’re going to do cost/benefit analyses,that will require a difference set of expertise. You are going to stick the FDA with a lot of stuff. They have enough trouble just doing the science. The risk we have if they did take that on that role is that it would create huge pressure on CMS to just to pay for any FDA approved drug. That pressure is already sufficiently high.
What are three main things we in the HD community can do to advance good science in spite of the systemic obstacles we’ve talked about, and strengthen our HD community?
Number one, you’ve got to support research. Our treatment of Huntington’s today is not adequate. It won’t get better without research, and research takes not only money but volunteers to participate. It also takes education of investigators and coordinators, so supporting high-level organizations and research is vital.
Two, don’t be passive. Get involved at the local level so you can teach each other and support each other. Because even if we don’t have a cure (which is not going to come tomorrow), we must help manage each other, and doing this takes a village.
Three, help each other within the family. It isn’t so much the chorea that impairs people and families but dealing with personality changes, agitation, shorter tempers… these are the things that cause people the most problems. These are the complaints from patients and families and what leads to job loss. Trying to help people through that family dynamic is so important.
Now we’ll end with the beginning. How did you become drawn into the HD community?
I became interested in movement disorders in my residency and have worked mostly with Parkinson’s and dystonia. I had had great neurology mentors, David Marsden and Fred Wooten, in particular. In 1985, I met the phenomenal Anne Young at a conference, and saw the basic science work she was doing in basal ganglia pharmacology. She, together with Iran Shoulson, started a group called the Huntington Study Group and asked me to get involved. So I did. It was really Anne Young who was my Huntington’s inspiration. Today, Washington University has a Huntington’s HDSA center, and while I work mostly with Parkinson’s patients and people with dystonia, I also see Huntington’s patients in our Center of Excellence.
Joel Perlmutter, M.D., is a member of the Huntington Study Group. At Washington University in St. Louis, he is Head of Movement Disorders and the Elliot Stein Family Professor of Neurology, Professor of Radiology, Neuroscience, Physical Therapy and Occupational Therapy, and also serves as Director of both the American Parkinson Disease Association Advanced Research Center for Parkinson Disease and the Huntington Disease Center of Excellence. He leads a research group that focuses on biomarker discovery, and studies of the pathophysiology of Parkinson disease, dystonia and related conditions. His research spans basic, translational and patient-oriented studies. He has had continuous NIH funding for 37 years and has published more than 350 peer-reviewed manuscripts.
