The Ocrevus PPMS ORATORIO study was designed for 630 patients. Eventually, though, Genentech ended up recruiting 732. You might think this is a good thing – the larger the study the better, right? Well, read on.
Note: much of the information in this blog post was taken from the CADTH Clinical Review of Ocrevus. CADTH is a Canadian government agency no one’s ever heard of that researches drugs to determine their effectiveness.
To ensure the integrity of a drug trial, the details of how it will be performed are published to regulatory agencies before the trial begins. This is call the study protocol. It should be obvious why this is important – if you’re allowed to continually make changes to an experiment while it’s in progress, you can probably tease any result you want out of it.
For example, imagine I wanted to prove that Houston was hotter than Dallas. I record temperatures every day for a month, but find that Dallas is hotter. I didn’t get the result I wanted, so, I simply extend my study by another month. Eventually, Houston has a heat wave, I conclude my study, and announce I’ve proved what I set out to prove. If, however, I were forced to declare the design of the study ahead of time (“The study will test the temperature at noon each day in the center of each city, in the shade, from January 1 to December 31…”), I would have far less room to distort my experiment to fit the result I’m looking for.
So it’s important to publish the protocol of a drug trial before it starts, and stick to that design. In the original protocol, dated June, 2010, Genentech very specifically described its patient population:
A total of 630 primary progressive MS patients will be enrolled and assigned
(2:1 randomization) to either an ocrelizumab [Ocrevus] arm or a placebo arm, stratified by age and region
Qualified patients are expensive to recruit. In fact, it took Genentech over a year to find its original 630 patient population. After reaching this goal, however, the company did a funny thing. It continued to accept patients, winding up with 732. Genentech attributed these additional patients to a “late screening boost”.
To report changes in the original protocol, drug companies file amended protocols with regulatory agencies. But, according to CADTH, Genentech failed to report the extra patients in its amended protocol.
Here’s why this matters: We know that with the 732 patients, Ocrevus was found to be – just barely – effective enough to merit FDA approval. 6.4% more of the 732 patients were helped by the drug than the placebo, resulting in a p-value of .03 for 12 week confirmed disease progression (CDP). Recall that the threshold for effectiveness is a p-value below .05 .
But Genentech provided another set of data to regulators, one you’ll never see it its advertisements. This data shows the study results if they’d followed the protocol, that is, with just that first 630 patients. The result? Ocrevus helped just 5.2% of people vs placebo. The p-value for this group was .09 – too large to be considered effective by regulatory standards.
In its clinical review, CADTH called out this irregular behavior:
The planned sample size for the ORATORIO trial was 630 patients; however, 732 patients were randomized due to an unexpected increase in screening after the closing of enrolment was announced by the manufacturer. This 16.2% increase in the sample size of the trial was not specified in a protocol amendment and had a measurable impact on the results of the study. A sensitivity analysis demonstrated that the primary end point of the trial would have not have met statistical significance without the enrolment of these additional patients.
There are a number of troubling possibilities here. By the time the 630th patient was recruited, the first patients had been in the study for over a year, and Genentech must have had initial data about the effectiveness of the drug – data which turned out to be less effective than its original protocol projection (30% and 43% 12 week CDP in the Ocrevus and placebo groups respectively). Realizing its drug was ineffective, did Genentech rush to round up a few more patients, hoping this might change the results?
And that brings us to the most disturbing part of the mystery patients – they received an extraordinary (some might say impossible) benefit from the drug versus the protocol group. Let’s look at the numbers. In the official study of 732 patients, 32.3% of those receiving Ocrevus experienced symptom progression, versus 39.3% of the placebo group:
| All 732 Patients, 12 week CDP | Ocrevus | Placebo | Total |
| Patients | 487 | 244 | 731 |
| Patients who progressed | 160 | 96 | 256 |
| Percentages | 32.9% | 39.3% |
Here’s the data from the 630 original patients – what the study should a have reported, had it stuck to protocol:
| Original 630 patients, 12 week CDP | Ocrevus | Placebo | Total |
| Patients | 420 | 209 | 629 |
| Patients who progressed | 145 | 83 | 228 |
| Percentages | 34.5% | 39.7% |
Subtracting the data in the second table from the first, we can get the results for of the 102 mystery patients:
| Extra 102 mystery patients, 12 week CDP | Ocrevus | Placebo | Total |
| Patients | 67 | 35 | 102 |
| Patients who progressed | 15 | 13 | 28 |
| Percentages | 22.4% | 37.1% |
Progression for the mystery patients in the placebo group was in-line with what you’d expect from the protocol population: 37.1% vs 39.3%. But progression for the Ocrevus group was 34.5% in the original 630 patients, versus just 22.4% in the mystery group. The patients in the mystery group pushed the p-value of the primary endpoint in the ORATORIO study over the effectiveness threshold, resulting in what will certainly be hundreds of millions of dollars in profits for Roche-Genentech.
What happened? We’ll probably never know. Genentech might just have gotten extraordinarily lucky. But anyone considering a marginally effective drug with serious side effects should consider just how many ways a drug company can tweak its results. If you’re ever trying to prove Dallas is hotter that Houston, or the other way around, the fine folks at Genentech may be able to help.
PPMS Blog 