Null hypothesis significance testing and reproducibility

Abstract

Although null hypothesis significance testing has been strongly criticized for decades, it has been the dominant statistical method in the field of psychology. Non-reproducibility of findings in psychology can be attributed, at least partially, to an arbitrary threshold (i.e., .05) in the null hypothesis significance testing and overrepresentation of p-values. The present study surveyed papers from the Japanese Journal of Social Psychology and examined whether or not such overrepresentation also existed among psychology researchers in Japan. Effect size measures and p-values did not correspond well when p-values were set at around .05. Moreover, the frequency of p-values just below .05 was greater than expected. These results imply that the overrepresentation of p-values can produce unreliable and irreproducible results. Two types of remedies are discussed to alleviate the problems of overrepresentation of the p-values.