Confidence intervals: read the uncertainty

Interpreting research data with uncertainty is a challenge. Many UX professionals misunderstand confidence intervals, leading to incorrect conclusions about design effectiveness or user preference. Understanding their frequentist nature is fundamental for evidence-based decisions.

The nature of the confidence interval

A confidence interval (CI) reflects the precision of your estimate of a population metric, such as a success rate. It does not indicate the probability that the true parameter lies within the observed interval. Instead, if you repeated the study many times, a certain percentage (for example, 95%) of the constructed CIs would contain the true population value. It's a statement about the method, not the fixed parameter.

Amplitude and basis of the estimate

Report the estimate, interval, calculation method, and eligible observations included in the calculation. Here, the base means how many and which observations support the result; it is not another name for the point estimate. Interval width depends on the data, observation count, and chosen procedure. A narrow interval does not fix recruitment bias or establish overall study quality. The repeated-coverage interpretation assumes that the conditions required by the method are met for the analysis.

Avoiding overlap misconceptions

In a hypothetical example, two sign-up flows have different estimates and partially overlapping intervals. That picture alone cannot establish equivalence or the absence of a difference. Compare the difference using a procedure suited to the study design and metric, and consider whether it would matter in practice. If the available observations or study design do not support the intended comparison, state that limitation instead of turning uncertainty into a conclusion that the two flows are tied.

Check your confidence intervals

• Understand that the CI reflects your estimate's precision, not the parameter's probability.
• Consider the interval's width as an indicator of your measurement's uncertainty.
• Do not conclude no difference merely from two intervals overlapping.

When analyzing your next research results, create a simple table with metrics, their point estimates, and respective confidence intervals. Use this table to discuss each measurement's uncertainty with your team, helping to make more realistic and informed product decisions.