A/B Testing Model & Prompt Changes

Marketing claims a new prompt, v2, "converts 20% better" than the current one, v1. Switching outright is a real gamble: if v2 is actually worse, that is real lost revenue for as long as it takes to notice and roll back. Proper A/B testing is how a team gets to trust a change like this before betting the whole product on it.

Route a portion of real users to each variant, prompt v1 vs. v2, or model A vs. B, using sticky assignment so the same user always lands in the same variant across their session. Measure a real success metric, conversion rate, satisfaction score, click-through, for each side. Declare a winner only once the difference is statistically significant, not the moment one number looks slightly higher than the other.

The scenario below is the exact one from the case brief: v2 genuinely does convert about 20% better. Only the sample size changes. Watch how much that alone matters.

How much traffic is actually needed depends on the traffic volume itself: a product with 2 million users can often detect a 2% difference within 24 to 48 hours, while a product with 10,000 users might need weeks to reach the same confidence for the same size effect. Knowing this before starting an experiment avoids the common trap of checking results early, seeing an exciting-looking but statistically meaningless number, and shipping on it prematurely.

The reverse risk deserves its own safeguard too. If a variant is performing significantly worse, not just slightly worse, early in the test, waiting out the full planned experiment duration to confirm it statistically is not caution, it is needless lost revenue. An automatic rollback trigger for a severe, early regression, the case brief's own guidance uses roughly 5% worse as the bar, protects against exactly that.