Did the fix work? How to measure customer experience improvement after an action

Why most fixes are never checked

Teams spend weeks deciding what to fix and almost no time checking whether it worked. To measure customer experience improvement, the check has to be planned before the change goes out, not reconstructed months later when someone asks why the same complaints are still there.

Step 1: choose the issue's own measure

If the fix was a second billing counter, the measure is the share of reviews complaining about billing waits at that store. If it was a clearer exchange policy, it is the share complaining about exchanges. The overall rating is a poor measure: it moves slowly, it includes everything else that happened, and on Google it can take up to two weeks to update after each review.

Step 2: record the baseline

• The issue's negative share in a fixed window before the change, for example the previous 90 days.
• The number of reviews in that window.
• The date the change went live, and exactly where.
• Two or three representative reviews, so you remember what the problem looked like.

Step 3: pick comparison stores

Customer complaints change for many reasons: seasons, prices, local events, network-wide policies. To see the effect of your change, compare the stores that got it with similar stores that did not. If billing complaints fell 6 points at the changed store and 5 points at its peers, the fix probably did little; if they fell 6 points at the changed store and rose at its peers, it probably did a lot.

When a fix rolls out everywhere at once, compare with the same months last year instead, and be more cautious about the conclusion.

Step 4: wait for enough evidence

A change needs time to show up in reviews: customers have to experience it, and enough of them have to write about it. Set the follow-up date in advance, typically four to eight weeks for a busy store, longer for a quiet one. Use the same window length as the baseline.

Step 5: decide

• Worked: the issue's share fell clearly more than at comparison stores, on a reasonable number of reviews. Keep the change and consider rolling it out.
• Did not work: no difference from comparison stores. Revisit the cause; the fix may have targeted the wrong thing.
• Not enough evidence: too few reviews to tell. Extend the window rather than guessing.
• Made things worse: the issue rose, or a new related complaint appeared. Read the new reviews before reversing.

Read the new reviews in every case. Numbers tell you whether the issue moved; reviews tell you why, and often reveal a side effect, such as faster billing but more errors.

A worked example

A jewellery chain adds a dedicated exchange desk at six stores from 1 March, after exchange complaints made up 12% of reviews there. Six similar stores without the desk are the comparison, at 11%. In April to June, the desk stores fall to 5% on 640 reviews; the comparison stores stay at 10% on 610. The gap opened after the change and held for three months: the desk worked, and the chain rolls it out.

Habits that make this routine

• Every action gets a baseline and a follow-up date when it is created.
• The monthly CX review starts with actions due for follow-up.
• Results are shared, including fixes that did not work.
• Changes to the taxonomy are noted, so before and after compare like with like.

Pitfalls

• Measuring too early. A week of reviews after a change is rarely enough to tell.
• Choosing the measure after seeing the results. Decide it when the action is created.
• Ignoring other changes. A price change or a festival in the same month can swamp the effect of your fix.
• Declaring success from the rating. A higher rating can come from anything; check the issue itself.
• Forgetting the comparison. Without stores that did not change, a seasonal dip looks like a win.

When you have few stores

With a handful of locations, comparison stores are hard to find and review counts are low. Lengthen the windows, compare with the same months last year, and lean more on reading the reviews: do customers still describe the problem, and do any mention the change? A smaller number of clear reviews saying "they've added a second counter, much quicker now" is real evidence, even if the percentages are noisy.

Doing this with Falcon

Falcon labels each topic in scope as needing attention, emerging or a strength, from its negative and positive counts and the change against the previous period. Each insight names the issue, why it matters (from the summary of the most-mentioned issue in scope) and a next step that names the taxonomy owner. Actions can be tracked to done, and because reviews keep arriving, the same signals show whether the issue's negative share falls afterwards.