April 2007

Making sense of the results from clinical studies

by Robert H. Blanks, PhD, President, Research and Clinical Science

Clinical decision making involves more than simply taking published results of research directly to the clinic. There are many factors to take into account ( e.g., patient preference, study value, practitioners training and experience, etc.) not the least of which is interpreting the results and incorporating worthwhile methodologies into the practice.

This article is a follow‑up to a previous series inspired by a presentation by Dr. Joseph C. Keating Jr. PhD. entitled "The Challenge." In his words, the "challenge" to the profession is "to determine the clinical meaningfulness (or lack thereof) of subluxation‑syndrome."

Dr. Keating states that "we've talked about it for more than a century... no one disputes the existence of subluxations, ...but the question has always been whether or not subluxations (or other segmental lesions) have health consequences (i.e., subluxation ‑‑syndrome)."

What is clinical meaningfulness? The focus of my previous articles was research and whether new areas of discovery have clinical meaningfulness ‑‑ whether the observed changes or differences mean something, or that they do not and should be ignored.

Once research has been generated, the next task for the profession is making sense of the results.

The usual statistical values from clinical research cannot be immediately applied to clinical practice.

Fortunately, there is a less‑often used computation that can solve the problems of interpreting and applying the results of clinical trials: the "Number Needed to Treat" (NNT).

This concept can also be used to express adverse events such as side effects, etc. A "Number Needed to Harm" (NNH) indicates the number of patients that must be treated on average to produce a given adverse event.

The NNT helps interpret the clinical meaningfulness of an intervention and quickly answers the critical questions:

1. Does the intervention work?

2. If so, how well does it work in comparison to groups receiving placebo, no treatment, or other interventions that are currently in use?

3. Is the new intervention safe?

Making sense of the published numbers

The results of clinical studies can be expressed in a variety of ways, but not all are helpful in making clinical decisions.

Typically, results are expressed in terms of risk. Risks are an expression of possible clinical outcomes and are probabilities that can vary between 0.0 and 1.0. If a certain risk has a probability of 0.0, it means that the event will never happen. If another risk has a probability of 1.0 it means that it always occurs.

For example, in a clinical trial with two groups of patients (experimental and controls), the experimental group might have a symptom rate of only 10% (risk of 0.10) whereas the control group has a rate of 30% (risk of 30%).

In this case, the treatment is said to be successful because it lowered the risk between the treatment (only 10%) vs. control (30%) group.

However, patients and their practitioners want to compare the findings across the treatment and control groups. This comparison is typically expressed as the relative risk (i.e., the level of risk in the treatment vs. control group), and is calculated by dividing the risk in the treatment group by that in the control group, 0.10 . 0.30, or 0.33.

Another typically reported value is absolute risk. Absolute risk reduction is determined by subtracting the risk in the intervention group from the risk in controls. The absolute risk reduction in our example would be 0.30 ‑ 0.10, which equals 0.20, or 20%.

Statisticians generally report relative risk to provide quantitative information on the effects of an intervention, and absolute risk to provide a number that gives the magnitude of the effect, i.e., a measure of clinical meaningfulness. Unfortunately, although reporting the absolute risk gets us closer to understanding the clinical meaningfulness of the study, the scores can be difficult to interpret.

This is why NNT scores are so powerful. The NNT provides information that is both quantitative and more readily understood by clinicians and patients.

How we calculate NNT  

The NNT for a given therapy is simply the reciprocal of the absolute risk reduction for that treatment. [1‑3]1‑3 In the example given above (where risk was 0.30 without treatment and 0.10 with treatment), the NNT would be 1 divided by (0.3 ‑ 0.1), or 5. In clinical terms, an NNT of 5 means that, on average, you would have to treat five patients to prevent the occurrence of that particular symptom in one patient.

In short, the NNT has direct applicability to clinical practice because it shows the effort required to achieve a particular clinical outcome, i.e., how many patients do I have to treat to achieve a given outcome.

Important qualities of the NNT

The NNT is treatment specific. It describes the difference between treatment and comparison group (in which patients receive placebo, no treatment, or some other treatment) in achieving a particular clinical outcome. A small NNT approaching 1.0, means that a favorable outcome occurs in nearly every patient receiving care and few if any of the patients in the comparison group. An NNT of 2‑5 indicates that a treatment is quite effective.

In some prevention trials, a NNT of 20‑40 could still be considered clinically effective. A number of clinical data sites are available to track NNT information; the Center for Evidence‑based Medicine at the University of Toronto is particularly helpful (http://www.cebm.utoronto.ca/).

Limitations in the use of NNT

Although NNTs are powerful statistical tools, they also have important limitations.

***  NNTs are condition specific and will differ depending on the patient's prior condition. On the other hand, if we have NNTs for different interventions for the same condition (and severity) with the same outcome, then it is appropriate to directly compare NNTs.

***  An NNT is always defined for a specified period of care. Only when the outcome is the same and is measured during the same period is a comparison valid.

***  An NNT is a single number, whereas the true value could be higher or lower depending on a number of patient‑ and study‑variables. To compensate for this variability, standard confidence (95%) intervals (CIs) are typically reported for each NNT. The upper and lower values of the CIs provide the statistical range within which the true value of the NNT falls 95 times out of 100. The larger the range of patient‑to‑patient variability (expressed as CIs), the more caution must be taken in interpreting and applying the study results.

How NNTs should be used

In biomedicine and in chiropractic, the distinction between therapy and preventative care is not always clear.

NNTs have been used extensively in both clinical applications, but the type of care (therapeutic, preventative) must be considered in interpreting NNT scores.

Thus, with therapeutic intervention (treatment), some form of therapy will almost always be necessary requiring careful consideration of the risks and benefits for each of the possible treatments.

In contrast, with preventative care the decision is doing nothing or doing something to prevent a bad outcome at some time in the future. The equation for preventative care also includes the possibility of harm without benefit for a considerable number of the patients.

Accordingly, clinically acceptable NNT scores are considerably lower for therapeutic interventions (where an intervention is required) than they are for preventative care where no intervention is a consideration and side‑effects may be of concern.

For example, in a therapeutic trial for treatment of head lice using Pediculicides (e.g., Permethrin) vs. placebo there is a nearly perfect NNT of 2(1‑2), meaning that the cure rate is one patient in two with a range of 1‑2 by following the recommended 14 day treatment protocol.[4] 4

In contrast, with preventative trials one could have clinically acceptable NNT scores of 15 (8‑12) during a three‑year trial of daily calcium and vitamin D to prevent (non‑vertebral) fractures in community dwelling seniors (> 65 years). [4,5] 4,5 Moreover, some still recommend prophylactic use of aspirin to prevent symptomatic deep vein thrombosis with hip replacement based on clinical trials reporting a NNT in this situation of 232 (140‑2239).[6]6

It is easy to see that choice of care depends on many factors. The point is that research employing clinical trials is required to provide the critical information that helps patients, physician and policy makers to know with reasonable assurance what to expect from treatment.

Moreover, with preventative approaches frequently employed in chiropractic, the issue will be to determine the benefit of intervention vs. the perceived risk. The NNTs for prevention tell us about the effectiveness for a population which is critical information required by insurance carriers and policy makers. Incidentally, chiropractic researchers use the acronym NNT even though chiropractic care is not "treatment" oriented. A more appropriate term would be "Number Needed to Adjust."

Research and clinical trials are the "change‑agents" in the health field. Evidence‑based approaches as being employed by Research and Clinical Science (RCS) and others in the profession, can improve the benefits of intervention and reduce the risks to patients. Rendering the results of clinical trials into one number (the NNT or NNH) assists the practitioner by providing a clinically relevant approach. However, NNTs are only one element of decision making process and must be integrated with patient preferences and the experience and judgment of the provider.

For me, one of the most compelling reasons for adopting the use of NNTs in chiropractic research comes from three major studies on health decision making. [7‑9]7‑9

In each of these, clinicians and policymakers were presented with the research results in different formats (NNT, absolute and relative risk reduction, etc.). In each case it was found that they made more conservative decisions when they received treatment effects expressed as NNTs rather than any other statistical measure.

As chiropractic continues to advance its evidence‑based culture to establish the meaningfulness of the chiropractic adjustment, we need to remember to report these findings in the most effective and meaningful manner. The NNT statistic appears to be the most effective in conveying the results of clinical trials to the broadest group of patients, clinicians and policy makers.

References

1. Laupacis A, Sackett DL, Roberts RS. "An assessment of clinically useful measures of the consequences of treatment." N Engl J Med. 1 988;318:1728‑33.

2. Cook RJ, Sackett DL. "The number needed to treat: a clinically useful measure of treatment effect." BMJ. 1995;310:452‑4.

3. McQuay HJ and Moore RA. "Using Numerical Results from Systematic Reviews in Clinical Practice." Ann Int Med 1997. 126:712‑720.

4. "Interventions for treating head lice." Cochrane Review, 14 Jan 1999. In: The Cochrane Library. Oxford:Update Software

5. Dawson‑Hughes B, Harris SS, Kroll EA, Dallal GE. "Effects of calcium and vitamin D supplementation on bone density in men and women 65 years of age and older." N Engl J Med 1997 337(10):670‑6

6. "Prevention of pulmonary embolism and deep vein thrombosis with low dose aspirine: Pulmonary embolism prevention (PEP) trial." Pulmonary Embolism Prevention (PEP) Trial Collaborative Group. Lancet 2000; 355:1295‑302

7. Naylor CD, Chen E, Strauss B. "Measured enthusiasm: does the method of reporting trial results alter perceptions of therapeutic effectiveness?" Ann Intern Med. 1992;117:916‑21.

8. Fahey T, Griffiths S, Peters TJ. "Evidence based purchasing: understanding results of clinical trials and systematic reviews." BMJ. 1995;311:1056‑60.

9. Bobbio M, Demichelis B, Giustetto G. "Completeness of reporting trial results: effect on physicians' willingness to prescribe." Lancet. 1994;343:1209‑11.

(RCS co‑founder and President Dr. Robert Blanks is Professor in the Department of Biomedical Sciences at Florida Atlantic University and a past Professor of Anatomy and Neurobiology at the University of California, Irvine. Prior to this he spent two years at the Max Planck Institute for Brain Research in Frankfurt, Germany and two years in the Department of Anatomy at Harvard Medical School. Dr. Blanks is on the Advisory Board of the International Spinal Health Institute, is a Board Member of the Council on Chiropractic Practice and is actively involved in chiropractic research. To learn more about health outcomes research and RCS, call 800‑909‑1354 or 480‑303‑1694.)