Why people try it

It is tempting to analyze reviews with ChatGPT or a similar tool: paste a few hundred reviews, ask for the main themes, and get a neat answer in seconds. For a first look at what customers talk about, that is genuinely useful. The trouble starts when the answer is used as data.

What goes wrong

  • Invented numbers: "35% mention service" may not come from any count.
  • Shifting categories: the same reviews produce different themes on different runs.
  • Truncation: long pastes are cut off, and the model summarises what it saw without saying so.
  • No evidence: the answer does not show which reviews support each theme.
  • Inconsistency over time: next month's answer uses different words, so trends cannot be compared.
  • Privacy: anything pasted leaves your systems; avoid including private customer data.

Prompting better

  • Give your categories with definitions and ask the model to choose only from them.
  • Ask for structured output: one line per review with category, sentiment and the exact supporting quote.
  • Work in small batches and count the results yourself.
  • Ask it to mark reviews it cannot classify as unclear.
  • Spot-check a sample against your own reading.
AI properly

How to use AI to analyse customer reviews you can trust.

How to use ai to analyse customer reviews

An example prompt

You will classify customer reviews. Categories: [list with one-line definitions]. For each review, output: review ID | category | sentiment (positive/negative/neutral) | exact quote from the review that supports the category. If no category fits, output UNCLEAR. Use only the categories listed. Do not summarise.
A structured prompt

Free review snapshot

Want this done for your own reviews?

No commitment. We use your details only to send your snapshot and follow up. Privacy policy

Where it still falls short

Even with good prompts, ad-hoc chat sessions do not keep a history, roll results up by store and month, catch model changes or let a store manager click from a number to the reviews behind it. Those are the parts that make review analysis useful week after week.

DIY limits

DIY review analysis with LLMs: what breaks.

An example

A manager pasted 300 reviews into a chat tool twice on the same day and got "long waits" as the top issue once and "pricing" the next. Rerun with fixed categories and structured output, and counted in a spreadsheet, the answer was stable: waiting was first, by a clear margin.

Common mistakes

  • Trusting percentages the model states.
  • Pasting more reviews than the tool can read.
  • Letting the model invent categories.
  • Pasting private customer data.

Common questions

Is ChatGPT accurate for sentiment?

Often reasonable on clear text; check samples, especially sarcasm and mixed languages.

Can it handle thousands of reviews?

Not in one chat; you need batching and your own counting.

Should we use it at all?

Yes, for a first look or drafting categories, with care.

What about privacy?

Check your provider's terms; paste only public review text.

Checklist

  • Fixed categories in the prompt.
  • Structured output with quotes.
  • Small batches.
  • Counting done outside the model.
  • Spot checks.

In short

Chat tools give a quick first read, but invent counts and shift categories. Use fixed categories, structured output with quotes, and your own counting; for ongoing tracking you need a consistent system.

Checking the output

Take twenty reviews from each batch and classify them yourself. Count how often you agree with the model on category and sentiment. If agreement is low for one category, sharpen its definition in the prompt and try again. Keep the prompt and category list versioned, so next month's results can be compared with this month's.

Doing this with Falcon

Falcon builds the theme list from your own reviews, then labels every review against it with a multilingual model, scoring sentiment for each mention and keeping the exact quote behind each label. Texts too short to judge are marked unclear rather than guessed, and low-confidence labels are flagged for a person to check. Negative share by topic, location and month then shows what is rising where, with small samples flagged.