Details

Autor(en) / Beteiligte

• Manapat, Patrick D.
• Anderson, Samantha F.
• Edwards, Michael C.
• Steinley, Douglas

Titel

A Revised and Expanded Taxonomy for Understanding Heterogeneity in Research and Reporting Practices

Ist Teil von

• Psychological methods, 2024-04, Vol.29 (2), p.350-361

Ort / Verlag

United States: American Psychological Association

Erscheinungsjahr

2024

Quelle

MEDLINE

Beschreibungen/Notizen

• Concerns about replication failures can be partially recast as concerns about excessive heterogeneity in research results. Although this heterogeneity is an inherent part of science (e.g., sampling variability; studying different conditions), not all heterogeneity results from unavoidable sources. In particular, the flexibility researchers have when designing studies and analyzing data adds additional heterogeneity. This flexibility has been the topic of considerable discussion in the last decade. Ideas, and corresponding phrases, have been introduced to help unpack researcher behaviors, including researcher degrees of freedom and questionable research practices. Using these concepts and phrases, methodological and substantive researchers have considered how researchers' choices impact statistical conclusions and reduce clarity in the research literature. While progress has been made, inconsistent, vague, and overlapping use of the terminology surrounding these choices has made it difficult to have clear conversations about the most pressing issues. Further refinement of the language conveying the underlying concepts can catalyze further progress. We propose a revised, expanded taxonomy for assessing research and reporting practices. In addition, we redefine several crucial terms in a way that reduces overlap and enhances conceptual clarity, with particular focus on distinguishing practices along two lines: research versus reporting practices and choices involving multiple empirically supported options versus choices known to be subpar. We illustrate the effectiveness of these changes using conceptual and simulated demonstrations, and we discuss how this taxonomy can be valuable to substantive researchers by helping to navigate this flexibility and to methodological researchers by motivating research toward areas of greatest need. Translational Abstract When replicating a scientific study, it is not reasonable to expect identical results - there will be some degree of variability from one study to another. However, too much variability between replication studies can begin to distort the truth and/or make it difficult to interpret a series of research results. Methodological and statistical choices that researchers make have the potential to add unnecessary variability. The many subjective choices involved in designing a study or analyzing data are likely to alter results, which increases the level of variability across a series of studies. Although progress has been made in addressing the role of researcher choice around methodological issues, the inconsistent use of terminology (e.g., researcher degrees of freedom, questionable research practices) has made discussions confusing. In this article, we present a new taxonomy for assessing research and reporting practices that is meant to clarify important terms and enhance conceptual clarity. We illustrate the usefulness of our new taxonomy with conceptual and simulated demonstrations and discuss how this taxonomy can be valuable to both substantive and methodological researchers.