To aid the communication of the various types
of experimental design, Alan and Will used a notation system presented in their
Figure 1. Such schematics have been attempted before, but I think the notation
used by Alan and Will has the great advantage that the arrows show exactly
which time-point is compared to other time-points for generation of the change
or difference scores. This favourable aspect of the notation system communicates
the precise link between experimental design and analysis of data.
Alan and Will included the time-series or
quasi-experimental design in their paper at my suggestion. Researchers might
wonder why such a design would be adopted at all, given its obvious lack of a
control group. One example might be situations in which time itself is the
intervention. Such is the case in studies on circadian variation in performance.
Any readers who have tried to research elite athletes might also find it
difficult or maybe even unethical to include a control group. In the future,
Alan and Will might like to extend their statistical expertise to this
situation in particular, since the analysis of time-series data might involve
complicated covariate analysis (to control for intervening variables also
changing over time) of correlated data-sets.
In the paper, two important issues were also
mentioned. First, even in a fully controlled trial, it was pointed out that
there may be reactive effects due to the participants knowing they have been
allocated to either the treatment or control group. Another so-called threat to
validity in a controlled trial is the potential for change in participants'
behaviour if they receive feedback about their pre-treatment scores before the
treatment or post-test. Even performing the pre-treatment test can in principle
affect the control and experimental treatments differently. Although any physiological responses to
exercise might not be due to such reactive effects, this threat to experimental
validity might influence an outcome measure of human performance. One design
that is supposed to estimate reactive effects due to the pre-treatment measurements
is called the Solomon 4-group. Using the new notation, the design is as follows:
The Solomon 4-group is a complicated design and
demands a large sample size (due to the inclusion of four groups). Nevertheless,
I have seen it employed in some large scale studies on physical activity
interventions, for example.
Secondly, the important issue of lack of
retention of research participants (often called subject mortality or
attrition) was mentioned by Alan and Will. If a treatment has been so badly
received by participants that they decide to vote with their feet, a researcher
can hardly label the treatment a success, even if the data analysed on the
remaining "selected sample" suggests that this is so! The CONSORT statement cited by Alan and Will
deals with this important issue by advising researchers to distinguish between
two types of analysis: intention-to-treat,
where you include all participants in the analysis, regardless of how well
they complied with the treatment, and as-treated,
where you include only those who did everything properly (Altman et al.,
2001). A good reference on the Web is
the intention-to-treat page at Gerard
Dallal's statistics site. Intention-to-treat analyses are an
issue where the outcome is mortality or morbidity that can be quantified
without a post-test, but exercise and sport-science analyses are mainly
as-treated, because participants have to get through the treatments and perform
the post-test before they can be included. Regardless, it
is important to document what happens to all the participants, and to justify
the approach you have taken in the analysis.
Back to article/homepage
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang L (2001). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine 134, 663-694
Published Dec 2005