Beyond Statistical Significance: Problems and Pitfalls of Quantitative Research

Abstract

Null hypothetical statistical tests reign supreme in research in most all disciplines when statistical analysis is used. While popular, novice researchers struggle designing, reporting, and analyzing quantitative studies. Five common problems in research are p hacking, HARKing, common method bias, sample and instrument bias, and assumptions not reported. Robustness means statistical assumptions must be evaluated and decisions made, as most assumptions will differ from theoretical recommendations. Moving beyond the concept of statistical significance and the misuse of p values, researchers can improve the link between statistical tests and results. After discussing the five common problems, a matrix is presented to guide researchers in improving study design to simplify and strengthen design. A major improvement could be adopting a continuum of practical significance with three factors.