Sentiment analysis accuracy: what to expect and how to check

Why accuracy varies

There is no single answer to sentiment analysis accuracy. It depends on how sentiment is defined (per review or per topic), the method (rules, trained models, language models), the language and the domain. Vendor figures measured on other data say little about how a method performs on your reviews.

The hard cases

• Sarcasm: "Great, another 40-minute wait."
• Mixed: "Lovely ring, terrible service."
• Comparisons: "Better than the other branch, still not good."
• Short: "ok", "fine".
• Code-mixed: "Staff accha tha but billing slow."
• Questions and facts: "Is parking available?"

How to measure it on your own data

1. Take a random sample of 200 to 300 reviews.
2. Have two people label topic sentiment independently, then agree on disagreements.
3. Run the method on the same sample.
4. Measure agreement by topic and by language, not just overall.
5. Repeat monthly on a new sample.

Note that people also disagree on some cases; their agreement rate is a useful ceiling.

What is good enough

For finding which topics are getting worse and where, a method that is consistent and close to human agreement is good enough, because trends survive small errors. For decisions about individual reviews, such as escalation, a person should check.

Improving accuracy

• Sharper category definitions with examples.
• Mention-level labels instead of review-level.
• An "unclear" option instead of forced labels.
• Requiring the supporting words for each label.
• Checking each language separately.

An example

A team measured its tool at high overall agreement, but much lower agreement on the billing topic. The cause was reviews like "billing was quick" labelled negative because "billing" often appeared in complaints. Adding examples of positive billing mentions to the category definition fixed most of the errors.

Common mistakes

• Trusting vendor accuracy figures.
• Measuring only overall accuracy.
• Not checking other languages.
• Never re-measuring after a model change.

Common questions

Checklist

• Own-data sample labelled by people.
• Agreement by topic and language.
• Monthly re-checks.
• Definitions improved where errors cluster.

In short

Accuracy depends on method, language and definitions. Measure it on your own reviews by topic and language against human labels, improve where errors cluster, and value consistency for trend analysis.

Questions to ask vendors about accuracy

How was accuracy measured, on what data and in which languages? Is it measured per topic or per review? How does the tool handle unclear text? Can it show the words behind each label? What happens to results when the underlying model is updated? Vendors who answer these clearly are usually more reliable than those who quote a single percentage.

Why an unclear option improves accuracy

Forcing every text into positive, negative or neutral pushes errors into the data. Allowing an unclear label for texts with no usable sentiment, such as "ok" or a single emoji, makes the remaining labels more trustworthy. Report the unclear share separately so you know how much of your feedback it represents.

Watching for drift

Accuracy can drift when models are updated, when new products bring new vocabulary, or when customers start writing differently. A monthly sample catches drift before it distorts a quarter of reporting.

Doing this with Falcon

Falcon measures sentiment for every theme a review mentions rather than one score per review, keeps the exact words that support each label, and reports negative share by topic, store and month with small samples flagged and unclear texts kept visible. Every number opens the reviews behind it.