Why it looks easy

Review analysis with AI looks easy in a notebook: paste reviews into a large language model, ask for topics and sentiment, get a tidy answer. The trouble starts when you run it every month, across every location, and people start making decisions from it.

1. Inconsistent labels

The same review can get different labels on different runs, and similar reviews can get different topics. Over months, trends reflect the model's variation as much as customers'. Fixing it needs a fixed topic list, constrained output and checks.

2. Drifting topics

Letting the model invent topics each time produces new names for the same thing: "slow billing", "checkout delays", "long queues at counter". Without a maintained taxonomy and merging, counts split and trends break.

3. Missing evidence

Labels without the customer's words cannot be checked. Managers stop trusting numbers they cannot verify. Requiring an exact quote for each label, and checking it exists in the text, adds work but makes results trustworthy.

Free review snapshot

Want this done for your own reviews?

No commitment. We use your details only to send your snapshot and follow up. Privacy policy

4. Short and mixed-language reviews

"Good", "ok", emojis and mixed-language reviews in Roman script are common. Models over-interpret very short texts and handle some languages less well. Short texts need a separate rule; languages need testing.

5. Scale and cost

Thousands of reviews per month across many locations means batching, retries, rate limits and a running bill. Re-running everything when the prompt changes multiplies the cost.

6. Maintenance

Models change, prompts are edited, topics need updating, new locations are added. Someone has to own the pipeline, test changes and explain why last month's numbers moved.

7. The product around it

Even with good labels, people need views by location and period, small-sample flags, access by role, and a way to open the reviews behind each number. That is a product to build and maintain.

Prompting pitfalls

Prompting an LLM to analyse reviews: what goes wrong.

Prompting an llm to analyse reviews what goes wrong
Build vs buy

Build vs buy review analysis with LLMs.

Build vs buy review analysis with llms

An example

A data team built a review classifier in a week. By month three, topic counts had jumped twice after prompt edits, regional managers disputed labels they could not check, and the analyst spent days each month fixing outputs. The team kept the prototype as a reference and moved to a maintained tool.

Common mistakes

  • Judging on a small sample.
  • No fixed taxonomy.
  • No evidence per label.
  • No owner for maintenance.

Common questions

Can LLMs classify reviews well?

Yes, with a fixed taxonomy, evidence and checks.

Is building cheaper?

Sometimes at first; ongoing effort often decides it.

When does building make sense?

With a dedicated team, unusual needs and long-term ownership.

What should we test first?

Consistency: run the same reviews twice and compare.

Checklist

  • Consistency tested.
  • Taxonomy maintained.
  • Evidence required.
  • Edge cases handled.
  • Owner for maintenance.

In short

DIY LLM review analysis breaks on consistency, drifting topics, missing evidence, edge cases, cost and maintenance. Each is solvable, but together they are a product to own.

Doing this with Falcon

If you are evaluating Falcon: it analyses Google Maps reviews of your stores and Google Play reviews of your app; classifies every review against a taxonomy built from your own reviews, with an exact supporting quote behind each label; finds and summarises specific issues; shows negative share by topic, store and month with small samples flagged; lets your team assign, draft and track replies, which are posted in Google Business Profile; and opens the reviews behind every number.