In any case, it can be concluded that several procedures for the analysis of repeated measures data show excellent control of the Type I error rate for normal data, even for extremely small sample sizes.

In this report, we investigated performance of the omnibus test using simulated data. The hierarchical procedure is a widely used approach for comparing multiple (more than two) groups.[1] The omnibus test is intended to preserve type I errors by eliminating unnecessary post-hoc analyses under the null of no group difference. However, our simulation study shows that the hierarchical approach is not guaranteed to work all the time. The omnibus and post-hoc tests are not always in agreement. As our goal of comparing multiple groups is to find groups that have different means, a significant omnibus test gives a false alarm, if none of the post-hoc tests are significant. But, most important, we may also miss opportunities to detect group differences, if we have a non-significant omnibus test, since some or all post-hoc tests may still be significant in this case.Although we focus on the classic ANOVA model in this report, the same considerations and conclusions also apply to more complex models for comparing multiple groups, such as longitudinal data models [2]. Since for most models, post-hoc tests with significant levels adjusted to account for multiple testing do not have exactly the same type I error as the omnibus test as in the case of ANOVA, it is more difficult to evaluate performance of the hierarchical procedure. For example, the Bonferroni correction is generally conservative.Given our findings, it seems important to always perform pairwise group comparisons, regardless of the significance status of the omnibus test and report findings based on such group comparisons.

The following ANOVA table illustrates the relationship between the sums of squares for each component and the resulting F-statistic for testing the three null and alternative hypotheses for a two-way ANOVA.

You will want to report this as follows:

Example: Reporting the results of ANOVA

Underlying Assumptions: Normality

Graphically, interactions can be seen as non-parallel lines connecting means when we are working with the simple two-factor factorial with 2 levels of each main effect (adapted from Zar, H. Biostatistical Analysis, 5th Ed., 1999). Remember interactions are referring to the failure of a response variable to one factor to be the same at different levels of another factor. So when lines are parallel the response is the same. In the plots below you will see parallel lines as a consistent feature in all of the plots with no interaction. In plots depicting interactions, you notice that the lines cross (or would cross if the lines kept going).

Step #3b

Step #3a:

Multiple comparisons: It is not good practice to test for significant differences among pairs of group means unless the ANOVA suggests some such differences exist. Nevertheless, I admit it is tempting to take another look at the comparison of G1 with G3 (ignoring the existence of G2 and perhaps assuming normality), but then you should use a Welch t test to account for the differences in sample variances, and you should not make claims about the result unless the P-value is as low as .01 or .02. Looking at that difference more carefully might prompt a subsequent experiment.

ANOVA Table

There are two tests that you can run that are applicable when the assumption of homogeneity of variances has been violated: (1) Welch or (2) Brown and Forsythe test. Alternatively, you could run a Kruskal-Wallis H Test. For most situations it has been shown that the Welch test is best. Both the Welch and Brown and Forsythe tests are available in SPSS Statistics (see our One-way ANOVA using SPSS Statistics guide).

However, platykurtosis can have a profound effect when your group sizes are small. This leaves you with two options: (1) transform your data using various algorithms so that the shape of your distributions become normally distributed or (2) choose the nonparametric Kruskal-Wallis H Test which does not require the assumption of normality.