Preference Test: Compare Alternatives with a Clear Question

When two proposals arrive at a product meeting, asking which one is preferred might seem the simplest way to end the discussion. A preference test helps gather this choice in an organized manner, but its usefulness depends on the criterion presented to the participant.

Choose a Dimension for Comparison

Imagine two fictional delivery tracking screens. One highlights the map; the other organizes the steps and estimated time. The team wants to know which presentation seems more useful for planning the receipt.

This question is more concrete than “which one did you like more?”. It informs the use situation and allows for analyzing reasons related to the need.

If the doubt is about aesthetics, ask about aesthetics. If it's about clarity, formulate a clarity criterion and consider complementing the choice with a comprehension activity.

Prepare Equivalent Alternatives

Keep content, state, and size comparable. If one version shows a complete order and the other uses incomplete data, the difference might be in the material, not the design.

Also review the labels under the images. “New version” and “old version” can suggest an expected direction. Use neutral identifications.

In UXTap, the preference block allows presenting two to four options, shuffling their order, and asking for the reason for the choice. The quantity should follow the project's decision; more alternatives increase the comparison effort.

Use Order as Part of the Design

The position of an alternative can influence the choice. Shuffling helps distribute this condition among participants but does not correct differences in size, prominence, or legibility.

Conduct a pilot on the intended device. Check if all images can be easily examined and if the person understands the criterion.

If the materials are too dense for a side-by-side comparison, consider another research design. The goal is to obtain a comprehensible evaluation, not to accommodate all proposals on one screen.

Read the Justification Along with the Vote

In the delivery example, someone might choose the map because they want to know if the driver is nearby. Another person might prefer the steps because they use the information to organize the concierge.

These reasons point to different needs. The team might discover that they need to combine information or offer a hierarchy that varies according to the moment.

Avoid reducing the reading to a percentage. Consult the participants' profiles and the reasons for the least chosen alternative. A relevant group might have a need that the majority does not share.

Differentiate a Clue from a Final Decision

If preference is divided, do not seek a winner at all costs. Check if the question was clear, if the proposals had relevant differences, and if the audience was coherently defined.

When the base is small, present the counts and the limits of the interpretation. If the decision depends on a quantitative difference, plan the sample and analysis before starting a new round.

Also avoid modifying the criterion after seeing the result. A version chosen for appearance cannot be announced as the most efficient without other evidence.

Prepare a Comparison That Doesn't Hide the Decision

On the fictional delivery tracking screen, the version with a map and the version with steps should represent the same order, at the same time, with equivalent information. If one shows an updated forecast and the other provides incomplete data, the choice may reflect the difference in content, not the way it is presented.

Write the context before the question: the person is trying to organize who will receive the order. Then, ask which alternative would help them most in this situation and why. “Which do you prefer?” leaves open whether the judgment is about beauty, trust, reading speed, or utility. An explicit dimension makes justifications more comparable.

Avoid calling the alternatives “old” and “new” or stating which one was designed by the team. These details can bring expectations that are not part of the evaluation. Neutral labels help discuss the material without turning the test into an approval of an internal project.

Use Justifications to Discover Choice Criteria

Create a table with the chosen alternative, main reason, and mentioned condition. In the example, one person might value the map because they will be leaving home; another might prefer the textual summary because they need to notify the concierge. The divergence suggests different use situations, not necessarily a quality problem in one of the screens.

Also look for justifications that do not match the material. If someone chooses the map believing it updates the location in real-time, but this promise is not defined, there is an expectation that needs to be investigated. Do not count this preference as validation of a functionality that has not yet been tested.

Read unfavorable reasons and cases without clear preference. A useful result might be discovering that both alternatives leave the time imprecise. In this case, redesigning the hierarchy without improving the offered information might preserve the difficulty. The research should open this possibility, even if the team expected to choose a winner.

What to Do with a Division of Responses

Before combining elements from the two versions, understand why each one was chosen. Putting the map, steps, all details, and all notifications on the same screen can increase complexity. A more specific hypothesis would be to highlight the receipt summary and allow consulting the map when it helps in the situation.

Compare the choice criteria among profiles you defined in the plan. Avoid looking for dozens of segments until you find a group favorable to the team's preferred alternative. If an unexpected difference arises, record it as a hypothesis for another investigation, especially when there are few responses in that segment.

The next activity should require the use of information. Ask the person if they can be absent for a certain period, using a fictional scenario with sufficient data. Check their interpretation, not just the vote. The guide on comparing versions helps separate an exploratory decision from a statement of measured improvement.

Take the Alternative to a Task

After understanding the reasons, select a hypothesis to validate in use. In the example, ask the person to identify when they need to be available to receive the order.

To get started in UXTap, prepare two equivalent options, write the context, and enable the justification. Review the presentation in the preview and conduct the pilot. The next step should verify if the declared reason for preference translates into a better experience for the task.

When the choice indicates an alternative, investigate your finding with first click. Check UXTap's preference feature to prepare comparable options and justifications.

Frequently Asked Questions about Preference

Should the most chosen alternative be published?

It can guide a decision, but the declared choice alone does not demonstrate performance in use. Consider the reason for preference, product requirements, and any misinterpretations. When the change affects an important task, test whether the chosen alternative allows it to be executed clearly.

Can I compare versions with very different styles?

Yes, if that difference is part of the question. Record which characteristics vary and avoid attributing the result to a single detail. If the goal is to decide the text of a button, keep the rest of the presentation similar to avoid unnecessarily mixing criteria.

Does shuffling eliminate all bias?

No. It helps distribute exposure to different positions but does not correct differences in content, size, or familiarity. The form of the question, the provided context, and the composition of participants continue to influence the choice. The test design requires attention to this set.

Create your UXTap account to prepare your first study.