Page 1 of 14http://spider.apa.org/ftdocs/ccp/1999/june/ccp673300.html 8/30/2000Methods for Defining and Determining the Clinical Significance of Treatment Effects Description, Application, and Alternatives Neil S. JacobsonDepartment of Psychology University of Washington Lisa J. RobertsDepartment of Psychology University of Washington Sara B. BernsDepartment of Psychology University of Washington Joseph B. McGlincheyDepartment of Psychology University of Washington ABSTRACTThis article summarizes and scrutinizes the growth of the development of clinically relevant and psychometrically sound approaches for determining the clinical significance of treatment effects in mental health research by tracing its evolution, by examining modifications in the method, and by discussing representative applications. Future directions for this methodology are proposed. Preparation of this article was supported by National Institute of Mental Health Grant 5KO2-MH00868-09. Correspondence may be addressed to Neil S. Jacobson, Department of Psychology, University of Washington, 1107 Northeast 45th Street, 310, Seattle, Washington, 98195. Received: August 24, 1998 Revised: December 7, 1998 Accepted: December 14, 1998 Jacobson, Follette, and Revenstorf (1984) proposed one of several methods for determining the practical importance of statistical effects found in clinical trials. In that article, as well as in subsequent publications ( Jacobson & Revenstorf, 1988 ; Jacobson & Truax, 1991 ), Jacobson and colleagues attempted to grapple with two limitations prevalent in statistical comparisons between groups of treated clients. First, such comparisons provide little or no information regarding the variability in treatment response from person to person. Group means, for example, do not in and of themselves indicate the proportion of participants who have improved or recovered as a result of treatment. Thus, statistical comparisons between groups shed little light on the proportion of participants in each condition who have benefited from the treatment. Second, standard statistical comparisons between groups seldom determine the practical importance of the treatment effects. Although previous investigators had tried to improve on standard statistical comparisons by reporting the size of the statistical effect (e.g., Smith, Glass, & Miller, 1980 ), even effect sizes do not directly speak to clinical significance. Although large effects are more likely to be clinically significant than small ones, even large effects can be clinically insignificant. Journal of Consulting and Clinical Psychology © 1999 by the American Psychological Association June 1999 Vol. 67, No. 3, 300-307 For personal use only--not for distribution.Page 2 of 14http://spider.apa.org/ftdocs/ccp/1999/june/ccp673300.html 8/30/2000There are a multitude of ways that one can characterize variability in treatment response and at least as many ways that one can determine whether changes are clinically significant (see articles in this special section). However the concept is defined, there are a variety of ways to operationalize clinical significance in mathematical terms. It is possible to distinguish the conceptual definition from its mathematical interpretation. Each has its own areas of controversy, and some of the mathematical debate is quite esoteric. Moreover, the mathematical distinctions between the original metric ( Jacobson, Follette, & Revenstorf, 1984 ) and more recent alternatives are all based on assumptions that cannot be tested without an empirical comparison of the methods. The ultimate question is "Which metric works best for a particular data set using a particular measure on the basis of some accepted external criterion?" Because such empirical comparisons have rarely occurred, in this article, we have chosen to concentrate on (a) defining our method, (b) examining its methodology, (c) providing examples of applications and misapplications, and (d) summarizing revised methods that retain our criteria for clinically significant change but proposing different metrics for doing so. We conclude with some recommendations for future research. Conceptual Definition of Clinically Significant Change Clinical significance is routinely defined as returning to normal functioning. Although for some disorders this may be too stringent a criterion, it is based on the assumption that consumers enter therapy expecting that their presenting problems will be solved. Even in cases in which this criterion is too stringent, the scientific community, as well as consumers of mental health services, still want to know how often normal functioning is attained. There is a second consideration: The magnitude of change for a given individual should be statistically reliable, that is, beyond the scope of what could reasonably be attributed to chance or measurement error. The final product is a twofold criterion for clinically significant change: (a) The magnitude has to be statistically reliable and (b) by the end of therapy, clients have to end up in a range that renders them indistinguishable from well-functioning people. If the client shows statistically reliable change but ends therapy still somewhat dysfunctional, then the client is classified as "improved but not recovered." If the client ends up in the functional range by the end of therapy, but the magnitude of change is not statistically reliable, then our method cannot determine whether or not the change is clinically significant. Finally, if the magnitude of change is statistically reliable and the client ends up within normal limits on the variable of interest, the client is said to have "recovered." This metric also allows for a determination of how often statistically significant deterioration occurs by identifying those clients who have shown a statistically reliable change in the opposite direction to that indicative of improvement. By applying our metric to a population of treated clients, one can determine the percentage of clients who improved but did not recover, the percentage of clients who recovered, and the percentage of clients who remained unchanged or who deteriorated in each treatment condition. These descriptive percentages can be compared between groups using contingency table analyses to determine whether any observed differences between groups are statistically significant, or they can simply be used descriptively to augment the standard between-groups comparisons based
View Full Document
Unlocking...