What’s Behind the Numbers? Important Decisions in Judging Practical Significance
11, 12-15, 2007 (sportsci.org/2007/ga.htm)
As he mentioned in his recent article (Hopkins, 2006), Will Hopkins' latest ideas about sample size estimation have arisen from a long-standing interest in the confidence interval approach to interpretation of study conclusions. Indeed, Will has been instrumental over the last two decades in communicating the advantages of such an approach amongst sport and exercise scientists. It is undeniable that confidence intervals help researchers to appraise the "real-world" relevancy of their study outcomes and that Will's spreadsheets are useful tools to help researchers make such an appraisal.
My personal interest in Will's article centers on the underpinning philosophy of the ideas rather than the mathematical accuracy of the spreadsheets derived from the "statistical first principles" which Will adopts. I know Will to be a highly competent mathematician who has a gift for communicating complicated mathematical concepts in a "researcher-friendly" way, especially through the use of his spreadsheets.
I think Will's claims that his new approach leads to sample sizes one third the size of "traditional methods" need to be viewed from a philosophical standpoint in order to unravel how this difference in numbers comes about. Such claims are especially interesting given that there are surprisingly tight relationships, both philosophically and mathematically, between some interpretations of the confidence interval approach and the null hypothesis testing process. For example, if the lower bound of a 95% confidence interval is exactly zero, then the exact P-value for statistical significance of the sample mean is 0.05 (5%). This makes sense, since both the lower bound of the confidence interval and the P=0.05 in the null hypothesis testing process basically suggest that it is unlikely that the true population effect size is zero (or, put another way, that the observed effect size is unlikely to be merely due to chance sampling error). I know that Will is not too comfortable with this relationship between 95% confidence intervals and statistical significance in the null hypothesis testing process and I believe this is one reason why 90% confidence intervals are preferred by him and other statisticians.
I would like to make some comments, which may be relevant, about the "null" in the null hypothesis testing process. Firstly, the null value does not have to be set at zero. The null assumption can also be that the effect size is equal to the smallest worthwhile magnitude. "Null" in this sense means "not important" and suggests that the null hypothesis testing process is not completely disconnected from issues surrounding practical significance. I think adoption of this philosophy in the past would have at least reduced the instances of researchers automatically assuming that statistical significance is synonymous with practical importance. It is also not very well known that, as part of the philosophy of a one-tailed, directional analysis, the null hypothesis should be stated that the observed effect is zero or opposite in direction to that hypothesized by the researcher. This is because both these scenarios should result in the same study conclusion; the intervention should not be adopted.
Given Will's claims, it may surprise some readers when I say that there are some published interpretations of confidence intervals (e.g., Guyatt et al., 1995) which lead to estimations of larger (not smaller) sample sizes than for the null hypothesis testing procedure (when zero is the chosen null value). This is because the lower bound of a confidence interval might be larger than zero (hence the sample mean is statistically significant) but might not be larger than the smallest worthwhile effect. Some statisticians interpret this situation as the sample size not being large enough to be reasonably certain that the true population effect is larger than the smallest worthwhile effect, i.e. more subjects are needed to narrow the confidence interval and therefore arrive at a more precise conclusion. One can tell from the work Will has done on boundaries of benefit/harm that he is one of the statisticians that does not agree with this rather conservative pass-fail approach to confidence interval interpretation. Still, it serves to illustrate that the interpretation of confidence intervals is itself under debate, even without bringing in the Bayesians!
So, in view of the drastic reduction in estimated sample size, what exactly is Will doing differently in terms of the philosophy of applying probabilistic statements to study conclusions? If multiple assumptions have been made, how have these been rationalized? The answer to this latter question is especially important given the oft-cited criticism that the popular P<0.05 (5%) cut-off value for statistical significance in the null hypothesis testing process is quite arbitrary, although to be wrong about a claim of significance, given the observed data, only one time out of 20 seems a decent delimitation of "reasonably certain" to me.
Will believes that the use of the P<0.05 cut-off value is not only arbitrary but it leads to decisions that are too conservative. Is Will fighting a generalization with another (or several other) generalization(s) in this respect? Who or what is P<0.05 too conservative for? Doesn't such a view actually detract from what is really important - that the level of alpha (or indeed any delimitation about probability coverage or levels in data analysis) is a situation-specific delimitation? The P<0.05 cut-off could be viewed as too liberal in some circumstances, e.g. the use of an antiviral drug to combat HIV infection when that drug might have serious side effects. Will's solution to this problem seems to involve the introduction of two new types of decision error with delimited acceptable cut-off values of 0.5% and 25% (to be fair, Will cites these as examples). What is the exact rationale for these values? Following these delimitations, then the acceptable cut-offs for qualitative conclusions of "beneficial", "trivial", etc, are introduced. What should these probabilistic values be and what philosophical basis drives them? If Will's new methods are adopted, then all these situation-specific delimitations should come to the forefront of the researchers mind. Do we need discussion-based position statements to be formulated for all these delimitations which affect the study conclusion process?
Inherent in the confidence interval approach to interpreting study conclusions is the most important delimitation a researcher needs to make; the selection of the smallest outcome magnitude that is clinically or practically important. Will maintains that any researcher who cannot arrive at such a value should "quit the field"! I can see his point in terms of the number of researchers who seem unable to even discuss the practical importance of their findings and agree that this inability is a terrible side effect of over-reliance on the null hypothesis testing approach. Nevertheless, I am not so sure that sport and exercise scientists have such an easy job in arriving at this smallest worthwhile effect.
Will maintains that a change of approximately 0.5 of the within-subject variability in performance between competitions is probably worthwhile for sports performance contexts (Hopkins et al., 1999). This cut-off value was arrived at following a study (the first of its kind) on the within- and between-athlete variability of real track-and-field performances at the elite level. Using these data, Will was able to estimate how much the within-athlete performance needs to change in order for it to make a difference in terms of winning places. But how does such a cut-off value relate to other scenarios, especially when such values have been calculated with all the variability associated with real-world situations? I am not challenging the delimitation here but wonder if we need to formalize the process of arriving at these decisions? Also, can such cut-off values derived from the real world be applied to the more tightly controlled environment of a laboratory experiment? For example, I have found recently that within-player variation (CV) of real soccer motion analyses can be as high as 100%. This variability is not surprising given the myriad of tactical and behavior variations between soccer matches. I don't think this magnitude of variability will be present if one researches an externally-valid component of soccer performance in the controlled environment of the laboratory. Will's value for a meaningful effect size of 0.5 x within-subject variability is at least better, in terms of underlying rationale, than Cohen's 0.2 of a between-subjects SD. How has this latter cut-off value been rationalized in terms of sports performance, physiology of exercise or indeed any outcome relevant to exercise science? Cohen was not a sport and exercise scientist, so he wasn't even in the field for him to be able to quit it!
Of course, the size of worthwhile effect should be an informed decision based on knowledge about what really makes a difference. But how easy is such a decision, especially when the study outcome variable is part of an overall concept? For example, what is the smallest difference in bowling speed that makes a difference to overall cricket performance of the team? This question was exactly the one Will needed to answer when he co-authored a recent paper (Petersen et al., 2004). In response to a training intervention, the smallest worthwhile change in bowling speed was stated by Peterson et al. to be 5 km/h as "the smallest that a top batsman would notice". Nevertheless, a smallest worthwhile effect size of 2.5 km/h was also stated as being "beneficial to a world-class bowler". As an illustration of how vital these decisions about smallest worthwhile effect are, and how clearly rationalized they should be, it was interesting that Peterson et al. found that the 90% confidence interval for the change in bowling speed was 1.2 to 4.2 km/h. This confidence interval tells us that a zero (null) change in true bowling speed is very unlikely (since the lower limit of the interval is 1.2). Nevertheless, the true change in bowling speed could be beneficial according to one delimited worthwhile effect (2.5 km/h) but not another (5 km/h), since the upper limit was higher than the former but lower than the latter delimited cut-off. Therefore, whilst Peterson et al. were pretty sure that the intervention induced an improvement in bowling speed, their study conclusion was less certain, according to their delimited worthwhile effect sizes. My question is to what extent should this ambiguity in the magnitude of the smallest worthwhile effect be built into Will's probabilities of "very likely beneficial", "trivial", etc? If the anchor between the delimited smallest worthwhile effect size and real world relevancy is pretty loose, is it actually worth being so precise with all the probabilities associated with the observed effect?
In summary, I believe that the most important issues in Will's article are not sample size calculations, but the new philosophy underpinning his new approaches to arriving at study conclusions using confidence intervals. There are new delimited conclusion error types and new boundaries of overlap between confidence interval and smallest worthwhile effect. Will has set a very important ball rolling but its path needs to be clearly steered and agreed on in my opinion.
Guyatt G, Jaeschke R, Heddle N, Cook D, Shannon H, Walter S (1995). Interpreting study results: confidence intervals. Canadian Medical Association Journal 152, 169-173
Hopkins WG, Hawley JA, Burke LM (1999). Design and analysis of research on sport performance enhancement. Medicine and Science in Sports and Exercise 31, 472-485
Hopkins WG (2006). Estimating sample size for magnitude-based inferences. Sportscience 10, 63-67
Petersen CJ, Wilson BD, Hopkins WG (2004). Effects of modified-implement training on fast bowling in cricket. Journal of Sports Sciences 22, 1035-1039
Published Dec 2007