It is estimated that 24.3 million people worldwide suffer from dementia and that annual costs for Alzheimer’s disease are as high as $155 billion in the United States (1996 US dollars).^1,2 One potential way to decrease the negative impact of dementia on people with this condition, on their families and on societies is to optimize the use of dementia medications,² with due consideration of both their effectiveness and their toxicity.

The effectiveness of most medications is tested via randomized controlled trials (RCTs). It is inevitable that some participants drop out of such studies before they are completed. Unfortunately, if analyses include only participants who remain in the trial, then study power is lost and erroneous conclusions may be generated. The principle of intention-to-treat (ITT) analysis, in which all patients are included in the analysis according to the group to which they were assigned at randomization, has become the accepted standard for the analysis of RCTs to try to counteract this problem.³ The strength of ITT analysis is that it not only preserves power but also promotes balance between treatment groups for both known and unknown confounders, thereby preserving the benefits of randomization.

Ideally, all possible data are collected on all subjects, including those who drop out of the study; however, this is not always possible. In order for ITT approaches to analyze all patients randomly assigned to a group, several methods to impute missing data have been developed.^3-10 Unfortunately, no statistical strategy can deal fully with all the different combinations of reasons for dropping out, dropout rates and different disease courses. At best, these techniques to impute missing data are educated estimates. One commonly employed technique to impute missing data is last-observation-carried-forward (LOCF), also known as end-point analysis.

LOCF substitutes subjects’ missing outcomes with the last measurement taken before they dropped out. It requires that 2 basic assumptions be met: the subjects’ responses would have been constant from the last observed value (i.e., the point at which they dropped out) to the end point of the trial; and, missing values are missing completely at random (i.e., dropout is not related to variables such as drug side effects, group assignment, disease severity or symptoms).^5-7

Authors have highlighted 3 factors that cause the second condition to be breached in a manner that introduces bias that will exaggerate the effectiveness of treatments as estimated by LOCF analyses; these include earlier dropouts or greater dropout rates in the treatment group and more rapid disease progression in subjects who drop out of the treatment group.^3,4,9-11 These factors result in more subjects who drop out of the treatment group having their decline artificially frozen at an earlier stage of disease, thereby potentially biasing results in favour of the drug under study (i.e., overestimating effectiveness relative to the placebo). By extension, study results may also be biased against the drug under study (i.e., underestimating effectiveness) if there are earlier dropouts or greater dropout rates in the control group or if there are subjects whose disease progresses more rapidly among those who drop out of the control group (Figures 1 and 2)

Since 1998, researchers have expressed concern that the use of LOCF in dementia drug trials contravenes the assumption of disease stability and the assumption of random missing data and hence risks generating biased results.^2,11-21 To better understand the significance of these concerns in dementia research we systematically reviewed the use of ITT and LOCF analyses, contraindications to the use of LOCF analysis, and the use of non-ITT analyses in RCTs of drugs approved for the treatment of Alzheimer’s disease, vascular dementia, mixed dementia and mild cognitive impairment in Canada (i.e., cholinesterase inhibitors such as donepezil, rivastigmine and galantamine, and the N-methyl-D-aspartate (NMDA) receptor antagonist memantine).

Figure 1. Differential last observation carried forward (LOCF) bias when there are more or earlier dropouts in the treatment group than the control group. [view]

Figure 2. Differential last observation carried forward (LOCF) bias when there are more or earlier dropouts in the control group than in the treatment group. [view]

We performed an electronic literature search of MEDLINE and the Cochrane Central Register of Controlled Trials from January 1984 (the year of publication of the McKhann criteria for Alzheimer’s disease²²) to February 2008 using the OVID search interface. The search strategy included the following terms: randomized controlled trials, dementia, Alzheimer, vascular dementia, mixed dementia, donepezil, Aricept, rivastigmine, Exelon, galantamine, Reminyl, memantine, Ebixa and cholinesterase inhibitor.

The principal investigator reviewed titles and abstracts to select an overly inclusive list of potential articles to be subjected to a full review of text and reference sections to identify relevant RCTs. The full text of selected RCT reports was then independently reviewed by 2 certified specialists in geriatric medicine with clinical expertise in dementia, formal research methodological training and recognized expertise in the review of dementia drug trials to determine which RCT reports met the inclusion criteria for the systematic review.

Of the 1146 articles identified by the search strategies, 191 papers (including RCT reports, reports of non-randomized trials, commentaries, systematic reviews and meta-analyses) were selected for full text and reference section review. Of these, 57 RCT reports met the eligibility criteria for systematic review (Fig. 3).^{2,14,20,21,26-79}

Contraindications to the use of LOCF as the only form of ITT analysis. Of the 34 studies employing LOCF as the only form of ITT analysis, 24 (71%) explicitly demonstrated contraindications (factors that could introduce bias) to its use. It was unclear whether the remaining 10 studies were free of contraindications, because most studies failed to report adequate data regarding the timing of dropouts and the severity of disease of the participants who dropped out. Consequently, Table 2 provides a range of potential contraindications to the use of LOCF for each study (the lower number representing the number of explicitly identified contraindications). Seven of the 57 trials in this review (12%) discussed the limitations of LOCF or non-ITT approaches.

Figure 3. Selection of studies for review [view]

Table 1. Characteristics of included studies (n = 57) [view]

Table 2. Types of ITT and non-ITT analyses employed and number of contraindications to LOCF analysis [view]

Table 3. Account of dropouts by drug class [view]

Table 4. Combined data for cholinesterase inhibitor and memantine trials [view]

Despite previously published cautions that LOCF analysis may introduce bias into dementia research, LOCF remains the most widely employed analytic technique in this research area; its results are rarely verified by other forms of ITT analysis. Further, the majority of the publications reviewed in the present study did not report the results of an ITT analysis, did not verify the results of LOCF with alternative ITT analyses when conditions that could introduce LOCF analytic bias in favour of the study drug existed, or did not comment on the results of alternative ITT analyses that were performed.

These problems were particularly evident in cholinesterase inhibitor trials. In the majority of these trials, either no ITT results were provided or LOCF ITT analysis was performed in the presence of contraindicating factors. For example, a higher dropout rate in the treatment group than in the control group was more common in cholinesterase inhibitor studies than in memantine studies (73% v. 25%), potentially biasing study results in favour of cholinesterase inhibitors and against memantine. Owing to a lack of data on the timing of dropouts and on the severity of disease in study participants who dropped out, our results may in fact underestimate the true prevalence of conditions promoting bias.

The concern that LOCF analysis introduces bias can be explored via ITT sensitivity analyses. If similar outcomes are generated when other forms of ITT analysis are employed, this provides some reassurance (but does not guarantee) that LOCF analytic bias does not alter results. Only 1 study verified the point estimates of efficacy calculated by LOCF analysis with an alternative ITT analysis.⁴² The 3 positive point estimates verified by alternative ITT analyses in this study are the only ones out of the hundreds of positive outcomes reported for LOCF analyses in dementia trials to have been verified in this way.

Some may erroneously argue that results of previous studies have been adequately confirmed by non-ITT analyses (i.e., techniques that exclude subjects without data from analysis), such as observed case analysis, completer analysis, fully evaluable population analysis or treated-per-protocol analysis. Like LOCF analysis, these non-ITT techniques may be systematically biased in favour of the group with greater, earlier or more severely affected dropouts and, consequently, they are not reliable, valid sensitivity analyses. The biases inherent in these non-ITT techniques have been highlighted by the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use,⁸⁰ by the European Agency for the Evaluation of Medicinal Products¹⁴ and by a number of authors.^15,53,67,81

Furthermore, the use of such non-ITT techniques and of LOCF analysis is completely unnecessary: other forms of ITT analysis that do not treat dropouts artificially by freezing values at the point of dropout but rather model for expected natural decline in dropouts could easily be performed either as the primary analysis or as a sensitivity analysis. The available approaches range from techniques that simply apply the rate of decline noted in the control group to all dropouts to more complex modelling procedures that are available in standard statistical programs. More appropriate forms of analysis have been employed in dementia research.^{42,50,52,56-58,68,78,79} Petersen’s study of mild cognitive impairment⁶⁸ may serve as a model for future research, as it employs both modelling for dropouts and sensitivity analyses of the effect of various modelling assumptions and approaches.

The present study cannot quantify the magnitude of the effect of the use of LOCF analysis on trial results; it is restricted to highlighting the high prevalence of conditions promoting bias in favour of more toxic therapies and against less toxic alternatives, such as memantine. As verification of results obtained using non-ITT and LOCF analyses requires individual patient data that are not publicly available, the onus is on the investigators who publish these trials to disprove the possibility that these analyses have introduced bias by performing ITT sensitivity analyses as performed in Petersen’s study of mild cognitive impairment.⁶⁸ This is particularly true for those studies demonstrating higher dropout rates in treatment groups.

These results are meaningful in day-to-day clinical care. Because this bias has likely exaggerated results in favour of more toxic therapies (e.g., cholinesterase inhibitors), this may have created inappropriate barriers to the funding and prescription of less toxic treatment options for dementia (e.g., memantine). Without accurate analyses, physicians cannot optimally counsel patients and families regarding appropriate therapies, and patients and families cannot provide truly informed consent when making treatment decisions. In addition, meta-analyses and pharmacoeconomic studies cannot be performed accurately and we cannot make reliable statements regarding whether trial results truly cross thresholds of clinical significance. These concerns, as well as the fact that LOCF analytic bias may prevent the funding and use of future less toxic treatments, should be of great concern to patient advocacy groups, such as the Alzheimer Society of Canada and the US Alzheimer’s Association.

In summary, it is highly unlikely, given the high prevalence of conditions promoting LOCF analytic bias in this study, that point estimates of some of the hundreds of positive outcomes generated in trials have not been affected in some way. The question is likely not whether bias been introduced, but rather the number of outcomes that have been biased and the degree to which they have been biased. As such, the present results provide empirical support for recent recommendations to researchers, licensing bodies and research guidelines bodies⁸² regarding their use of LOCF analysis. One of these recommendations is that the CONSORT group (www.consort-statement.org) consider incorporating guidelines regarding appropriate analyses for studies of medications used to treat chronic progressive disorders into the CONSORT statement so that journal editors, funding agencies, ethics review boards and drug formulary committees can request that these recommendations be followed in future studies of dementia and other chronic progressive disorders. In the meantime, researchers should ensure that analyses promoting bias are avoided or scrutinized using alternative ITT sensitivity analyses. Further, licensing agencies (e.g., the US Food and Drug Administration, the European Agency for the Evaluation of Medicinal Products, and Health Canada) should review this situation immediately to determine whether they will continue to accept LOCF analyses in research on dementia and other chronic progressive conditions.