Extreme times: investigate before excluding

When analyzing usability test data, excessively long task completion times can appear as anomalies. The temptation to simply remove these "outliers" is strong to avoid skewing averages. However, discarding them without investigation risks losing the chance to identify critical usability problems or unexpected technical failures.

Why long times matter

A long duration may reflect interface difficulty, a pause, a technical problem, or the way the clock was recorded. The value alone cannot distinguish those explanations. Check the measurement’s start and end, then review the available session evidence. If there is no recording or sufficient log, mark the cause as unknown. Do not automatically turn an extreme value into an error, fraud, or confirmed evidence of usability difficulty for the person who participated.

Strategies for investigation

Before any exclusion, dive into the specifics. Watch user session recordings, review comments, and observe patterns. For example, imagine a hypothetical user takes 10 minutes for a sign-up process designed for two. The recording might reveal they repeatedly tried to enter a complex password without clear feedback, or that the "next" button was unresponsive. This points to a design or technical issue, not a "slow" user.

Careful analysis and reporting

Always retain the original raw data. When reporting, consider sensitivity analysis: present results both with and without extreme data, explaining your rationale. The median can be more robust than the mean with extreme data, but it doesn't tell the whole story. Also use range and quartiles to provide a more comprehensive view of time distribution, allowing inspection of the actual variability.

Checklist for extreme times

• Review session recordings for users with prolonged task times.
• Check user feedback and system logs for potential failures.
• Compare metrics with and without extreme data for sensitivity.

Create a team protocol for investigating extreme task times. This document should detail steps for reviewing recordings, logs, and feedback, and how to document reasons for retaining or excluding data, ensuring evidence-based decisions.