Problems with multiple comparisons

In statistics, multiple comparisons occur when both a series of statistical inferences are taken into account or a subset of the parameters selected based on the observed values (multiple comparisons problem).

Cause: When a person estimates a subset as a whole, wrong inference is likely to occur, including that the confidence interval does not contain the corresponding overall parameters or that the hypothesis test incorrectly rejects the 0 hypothesis. In this respect, I would cite two examples as a description.

① Suppose we want to judge a new method of writing teaching and the quality of the traditional method. Then we divided the students into two groups, one group using the new Method (treatment group), one group using the traditional method (control group). We can evaluate the performance of the two groups according to the grammar, spelling, content, etc. of the students, but with the increase of the evaluation parameters, because of the stochastic factors, the two groups will behave differently at least one parameter, then how do we judge whether this is a random performance or the use of the method?

② Classic Coin toss problem: a uniform coins (i.e. both positive and negative probabilities equal) throw 10 times at least 9 positive probability is (10+1) *0.5^10 = 0.0107, this probability is less than 0.05, so we can generally be presumed a coin toss in the experiment this situation will not happen, conversely, If this small probability event occurs in an experiment, you can claim that the coin is uneven. Now, suppose we have 100 evenly-spaced coins and put them at the same time as the trial of the appeal, not a coin toss 10 times at least 9 times the positive probability is (1-0.0107) ^100=0.34. From this, we are likely to mistakenly infer that at least one coin is uneven. So the judging criteria of a single coin are not applicable to multiple coins.

Workaround: in order to solve the appeal problem, you can use the error detection rate (Fdr:false Discovery rates), Bonferroni correction and other methods.

Note: Use the following URL to summarize: Https://en.wikipedia.org/wiki/Multiple_comparisons_problem

　　

Problems with multiple comparisons