Dryer’s paper is particularly insightful for the purposes of this study because it proposes a new sampling technique to control for genetic and areal bias in order to estimate a so-called “adjusted frequency”. In other words, he aims at estimating the expected frequency of each possible word order after controlling for genetic and areal correlations. The present replication study will focus on this adjusted frequency count.
The author reused the case study of Dryer (2018) word order pattern.
Since the purpose of this paper is to gauge the effect of the particular method used for analysis on the results, we use the original data without additional modifications for all four case studies. Therefore, we will not be concerned with questions regarding the particular choices made by the authors in the data collection and annotation for the original studies. Our purpose is not to contest the linguistic work of the papers in question, but simply to check the original results against a different statistical technique. More specifically, we will follow Guzmán Naranjo and Becker (2022) and Verkerk and Di Garbo (2022) in using phylogenetic regression to control for genetic effects and a Gaussian Process to control for contact and areal effects (cf. Section 3). As we will show in Sections 4, 5, and 6 for three test cases, some findings are robust and can be corroborated with our methods, while others cannot be confirmed. This underlines how important it is to be aware of statistical methods having an impact on the results as well; they need to be chosen with as much care as the linguistic choices concerning the dataset and annotation, and they need to be reported with transparency to allow for evaluation and replication. We discuss this in more detail in Section 7.
This is a major methodological advance for typology. These models require statistical expertise, which may limit accessibility for many linguists.
Replication and replicability are fundamental tools for ensuring that research results can be verified by an independent third party, reproducing the original study and ideally finding similar results. If so, then, more certainty can be attributed to the results due to cumulative evidence. Thus, replication serves the purpose of consolidating the findings, as they are arguably more robust when being reproduced
The authors made a clear distinction between replication, and replicability. This denotes that replication is not just re-running the same test, but as testing the dependence of conclusions on analytic choices.