Open-ended responses with AI: organize themes without losing people's voices

Open-ended responses reveal details that a scale doesn't predict: an expectation, an unknown rule, or a use case the team hadn't considered. When the volume grows, AI can help organize this material. The caution is not to let organization replace interpretation.

Prepare a question that generates useful material

Imagine a fictional expense tracking product. After a task, the team asks “what made it difficult to record this expense?”. The answer might mention a field, a document, or a question about the process.

Compare this with “any comments?”. A broad question might be suitable in some contexts, but it offers less guidance for someone who needs to report a specific experience.

In UXTap, open-ended questions can have mandatory settings and minimum response criteria. Use these options to support participation, without assuming that more characters mean a better contribution.

Differentiate cloud, sentiment, and theme

A word cloud highlights terms present in the material. Sentiment seeks to classify the tone of a response. Theme organizes the subject matter. These are distinct layers.

A frequent word like “card” can appear in compliments, questions, and difficulties. It alone doesn't explain what needs to change.

In UXTap, open-ended analysis combines word cloud and, when AI is available in the plan and operation, sentiment and categories. Automatic updates depend on conditions such as new responses, processing interval, and usage limits. It is not correct to promise immediate classification of every response.

How to analyze open-ended responses with AI

Open a sample of responses from each group and check if the name represents the content. Look for overly broad categories, overlapping groups, and comments that don't fit well.

In the expense example, “submission problems” might mix difficulty attaching an image with a question about who the request will be forwarded to. The team will need to decide if these subjects should remain separate.

In UXTap, categories can be renamed and merged. Conduct this review with a recorded criterion so that the taxonomy remains understandable in the next round.

Treat classifications as reading support

Sarcasm, short responses, and phrases with mixed evaluations make automatic interpretations difficult. Read ambiguous cases and do not use a sentiment label as a diagnosis of the person.

A response like “I succeeded, but I had to try again” contains a favorable outcome and difficulty. The product decision depends on the episode, not just the attributed tone.

UXTap verifies if examples used in category creation exist in the received material. This helps with traceability, but it doesn't guarantee that every category represents the best possible interpretation. Review remains necessary.

Preserve the denominator and exceptions

Report which responses were analyzed and which were excluded from a summary due to lack of useful content. Do not transform a frequent category into a percentage of all customers when the collection came from a specific segment.

Also look for rare comments with relevant consequences. A rarely mentioned problem can block an essential task. Frequency and impact need to be discussed separately.

Create a small analysis code for expenses

In the fictional expense product, imagine three comments: “I didn't know which receipt to send”, “the file was not accepted”, and “I didn't understand who receives the request”. All three might mention submission, but they point to different difficulties: information about the document, execution of the attachment, and understanding of the process.

Prepare a card for each theme with a name, definition, included examples, and situations that should be excluded. “Attachment refused” might include error messages and file incompatibility, but not a question about the approver. This delimitation makes it easier to review a classification and explain why two categories were kept separate.

Use fictional examples to align the team before reviewing real data. Then, replace abstract discussion with reading the collected comments. A response might address more than one subject; if your analysis requires assigning a main theme, record that criterion instead of pretending other themes don't exist.

Conduct a distributed review, not just of the clearest examples

Start with responses from large, small, and poorly defined themes. Include short, long, ambiguous texts, and those with mixed sentiments. Checking only obvious examples can hide errors precisely where classification requires more context. Pay special attention to themes that will support a product decision.

If two people review the material, discuss discrepancies with the original open-ended response. The goal is not to defend the first label, but to make the rule understandable. A category that always depends on the intuition of its creator will be difficult to reuse in the next round.

Record relevant taxonomy changes. Merging “documents” and “attachments” might improve organization or erase a distinction between a rule and a technical failure. Before changing, check the content of the groups and how the decision will affect comparisons with previous analyses. The number of categories is not a measure of quality in itself.

Present frequency without erasing meaning

Explain whether the count represents responses, people, or mentions. A person who writes an extensive comment should not gain more weight just by repeating a word. If multiple themes can be attributed to the same response, the totals per category might exceed the number of responses; this needs to be clear in the presentation.

Also consult exceptions that challenge the dominant theme. If many comments mention attachment difficulty, but some describe easy submission followed by a question about approval, there's a subsequent step to investigate. A less frequent theme can be relevant due to the effect it has on the task.

Conclude the analysis with a verifiable question. For “I didn't know which receipt to send”, the next investigation might evaluate whether document examples help choose the correct attachment. Use a content test to examine this understanding. AI helps organize the material; the recommendation should remain linked to evidence that another person can review.

Bring the theme to the product

Choose a category, consult its responses, and write an observable hypothesis. If the problem is attaching receipts, test that task and see what happens.

To start, add a specific open-ended question after an activity in UXTap. When material is available, review the themes and save examples with context. The next action should link the group of responses to an investigation or change that the team can evaluate.

Save the reviewed themes with context in the research repository. The results resources help relate the synthesis to the supporting material.

Frequently asked questions about open-ended responses and AI

Can I present the automatic synthesis as a participant's speech?

No. A synthesis is an interpretation or condensation of the material, not a literal quote. Identify what is a summary and preserve the original text when you need to present a quote. Verify that cuts and context maintain the meaning before using the excerpt in a decision.

Does negative sentiment mean a usability problem?

Not necessarily. The response might address price, service policy, expectation, or an external situation. Read the content and relate it to the activity performed. The tone helps organize a reading, but it doesn't automatically identify the cause or the priority of the problem.

When should I redo the categories?

Reevaluate when new subjects don't fit the definitions, when categories overlap, or when the analysis objective changes. Preserve the history of the rule used in each round. A taxonomy review might be necessary, but it should not appear as a change in public behavior without explanation.

Create your UXTap account to prepare your first study.