Statistical Problems in Papers in the Journal of Japan Society of Acupuncture

Abstract

In investigating all the back articles of the Japan Society of Acupuncture Journal (vol. 32 no. 1 1982) until the present (vol. 41 no. 2 1991) it was found that out of 169 articles using statistical analysis 164 of them (97%) contained some kind of problem. Of these, 99 out of 105 (94%) stimulus experimentations by animal and human, and 62 out 64 (97%) clinical studys were found to contain misuses of statistics approximately the same percentage for each group.
A high incidence of problem was of multiplicity, with 73 out of 164 problem articles (45%) containing misuse of statistics.
Beside this, 19% (31/164) of the articles failed to list the methods of analysis or test of significance used, 13% (22/164) published their conclusions based only on comparisons within a group, 14% (23/164) published only the rate of effectiveness, without reporting the rate of uncertain results, for example, confidence interval, and 12% (19/164) failed to give assurances against randomness. Overall there are 18 checkpoints, but there is a less than 10% rate of statistical misuse in each of the others.
Because each report has multiple problem the total percentage is greater than 100%. Even excluding the 19% of the articles failing to list methods of analysis or test of significance and the 14% listing only the rates of effectiveness, neither of which ultimately affect the article results, there ara still statistical misuse in 62% of the articles using statistical analysis.
Looking at the authors it can be seen that 11 writers account for 30% (49 articles) of the problem. In short, the same people are repeating the same mistakes in from 3 to 7 articles each.
Of these, 91% contain problems of multiplicity, and 55% contain comparisons conducted within a group instead of between groups as required. These are their most glaring faults.