Purpose of Studies Evaluating Therapy Effectiveness

The aim of studies evaluating therapeutic effectiveness is to evaluate how well a new therapy compares to standard treatment with respect to one or more outcomes of interest. In these types of studies the first goal is to minimize systematic differences between treatment and control groups except with respect to the treatment itself. It is important to note that randomized controlled trials allow investigators to avoid most sources of bias except those that might affect generalizability, or external validity, of study findings. Consequently, even RCTs may have systematic biases that might limit their use in clinical situations.

Common sources of bias in therapy studies

1. Selection (sampling) bias—Who is in the study?

This problem is similar to that found with diagnostic studies and tends to affect the generalizability (external validity) of the study to other populations. Consequently, if the study population is not representative of most clinical situations then the results of the study may be somewhat suspect.

2. Spectrum bias—How sick are the subjects in the study?

This problem is similar to that of diagnostic studies but differs in an important way. In diagnostic studies the spectrum of the “diseased” (i.e., non-control) group is narrowed to extremely sick subjects who are then compared to the control group. On the other hand, for therapeutic studies the spectrum of the entire study group may be important regardless of whether the study is observational or randomized-controlled. This is true because the new treatment may work differently depending upon the severity of patient disease.

3. Failed allocation concealment—Do the patients and/or investigators know the treatment?

When patients, investigators, or treating physicians are aware of which treatment arm to which a patient is assigned then the very purpose of randomization becomes somewhat compromised. This is why blinding subjects and investigators, whenever feasible, is of great importance with RCTs. It is worth noting that in some situations, blinding is not possible. For example, if one treatment is surgery and the other involves medical therapy it is not possible to conceal the treatment from patients or treating physicians, due to ethical issues and patient safety concerns. Nevertheless, when possible, patient allocation in RCTs should be concealed.

4. Withdrawal bias—How many subjects quit the study and why?

When too many subjects withdraw from the study then there may be a systematic withdrawal bias that could affect the generalizability (external validity) of the study to typical clinical populations. It might be asked, Why did these patients withdraw? Was there some characteristic that these patients have in common?

5. Analytical bias—What if subjects decide to switch treatments?

If subjects who later decide to accept another treatment are analyzed with the group to which they have chosen then analytical bias may be introduced. For example, if a subject randomly assigned to allogeneic blood decides to bank autologous blood for surgery then a favorable outcome may derive from the motivation of the subject for the new treatment rather than from the treatment itself. Most good randomized controlled trials use an “intention to treat” analysis, in which subjects are analyzed with the groups to which they were originally assigned, to avoid this type of bias.

6. Measurement bias—How were outcomes measured?

If outcomes for those receiving the new treatment are measured or recorded differently than outcomes for those in the control group then measurement bias may be introduced into the study.

7. Referral bias—What types of hospitals or clinics were used as study sites?

If the study is conducted at large tertiary care centers then there is a possibility that only more difficult (higher disease severity) cases will be included in the study. Furthermore, greater resources at a large referral center may make findings appear more optimistic than they might be in other settings. Both of these factors may limit the external validity of study findings when applied to smaller hospitals or clinics.

8. Lead-Time bias—Was the disease diagnosed earlier in one group thereby leading to improved survival?

This applies mostly when survival is one of the measured outcomes. If the treatment is tied to a diagnostic test that permits one to establish a diagnosis earlier than is typical then the survival in that group may be longer solely as the result of earlier diagnosis (earlier spectrum of disease) and not the treatment per se. This becomes a greater issue in prognostic studies than in therapeutic studies.

9. Self-selection (volunteer) bias—Did subjects differ significantly from the population because they volunteered to participate in the study?

Recruitment of volunteers to participate as study participants may also limit the generalizability of study findings since there may be characteristics of those who would be willing to volunteer for a study that differ from those of most clinical populations. To some extent this can be resolved by carefully matching volunteers for demographic and clinical features found in the population for which the treatment might be indicated.

References

Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM: The relationship between methodological quality and conclusions in reviews of spinal manipulation. JAMA 1995;274:1942-1948. [PubMed]

Greenhalgh T. Assessing the methodological quality of published papers. BMJ 1997;315:305-308 [PubMed / Free Full Text]

Guyatt GH. Sackett DL. Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA 1993;270:2598-2601 [PubMed]

Guyatt GH. Sackett DL. Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA 1994;271:59-63 [PubMed]

Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, Contopoulos-Ioannidis DG, Lau J. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA 2001;286(7):821-30. [Free Full Text]

Ioannidis JP, Karassa FB. Bias in uncontrolled therapeutic trials in rheumatology due to selection of populations with extreme characteristics. J Rheumatol 2001;28:1881-1887 [PubMed]

Liberati A, Himel HN, Chalmers TC: A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986, 4:942-51 [PubMed]

Phillips C, Tulloch JF. The randomized clinical trial as a powerful means for understanding treatment efficacy. Semin Orthod 1995;1:128-38 [PubMed]