Definition Statistical significance

We will identify the result of a statistical calculation as significant if the probability of error that an assumed hypothesis also applies to the population does not exceed a previously determined threshold. To put it more simply: A measured relationship between two variables in a sample is not random but applies to the population as well. We can only test the significance of hypotheses, not the results of individual variables. The answer to/result of the question "What is your weight?" cannot be tested in terms of significance – the variable has to have a relationship or dependence with at least one other variable.

An example: If we compare the variables body weight and height, we will likely identify a statistical dependence between the two, and in this particular case it will probably be a positive correlation. The term 'positive correlation' translates to the hypothesis that the increase of the value of one variable is associated with an increase of the value of the other variable (in this case, =more body height equals more body weight and vice versa).

The crucial question: does this dependence, which is true for the sample, also apply to the population, or does the result of the sample represent a random result? In order to find out, we have to determine the threshold value of the error probability (p-value) for our hypothesis (in this case, the positive correlation). The significance level (α) indicates the threshold of the probability of error. Traditionally, the significance level is 5 percent, i.e. α = 5%.  Next, we have to conduct a hypothesis test and apply it to the variables at hand. The result of the test states the p-value, i.e., the probability of error. If this p-value is under α = 5%, the result is considered to be significant.

So, if we have proven a statistical relationship such as our hypothesis about the dependence of body height and weight to be significant, this means that the dependence of the sample also applies to the population with a probability of 95%. Thus, there is still a residual 5% chance that the dependence is due to chance. This would be relevant for 1 in 20 cases.

Significance is not to be confused with the margin of error, which indicates the deviations (in percentages) of individual results from the actual results of the population.