Lately I’ve been getting messages on LinkedIn from salespeople with a “revolutionary” idea — let’s measure the sentiment and topics of user comments and then improve retention based on that.
What makes it especially funny is that this is exactly the topic that formed the basis of Anya’s finished master’s thesis — and my unfinished one — eight years ago. Long story short: what people say and how you can influence them are almost unrelated. And even what people do and how you can influence them aren’t that closely connected either — something I learned firsthand three years ago when I tried to retain users who were determined to leave the game. As determined as they were to leave — they left, despite all the “good” we tried to do for them.
Time to recall my usual mantra: no, you don’t need AI for analytics, and there’s a very simple reason. A huge chunk of the insights AI will “surface” for you are so obvious that instead of integrating yet another trendy tool into your product or your social media, you could just put on the user hat and look at your own product from under its wide, lovely brim.
Let’s say you actually did this two-hour exercise but still don’t trust yourself. Fine — go to the Facebook comments and read what people write there. Go to customer support and read what people write there. Not AI-generated summaries with nicely highlighted text: “Users complain that items are too expensive!”
Diamonds in this game cost a fortune. Devs, have some shame!
— Alice
The Berserker bundle is $11 now, used to be $8. Stopped buying it.
— Bob
Come on, this is nonsense. What exactly do you get from that AI report that you wouldn’t get from looking at the comments with your own eyes? In most cases, if you ask your support agent or community manager, or if you spend a couple of hours reading yourself, you’ll see the same 3–4 biggest issues in the game. Do you really have so many resources in the company that you can address more than 3–4 issues at once — and need AI just to speed up choosing between problems mentioned in 1.4% of comments versus 1.8%?
And then it gets even funnier: “let’s determine which group a user belongs to and give that group special offers.” Sure! But then let’s also give the exact same special offers to users who don’t belong to that group. Oopsie-doopsie, segmentation-kookareekoo, cluck-cluck-cluck. Any segmentation — any at all — has to outperform not just the baseline, but also the opposite version. Meaning: it has to beat not just 00..00 but also 11..11, while representing the correctly identified users who remain zeros. That is ALWAYS a harder task. You don’t have the resources, and you don’t have the product depth to make AI-driven segmentation worth it. My favorite example: “predicting likelihood to pay,” which is about 95% determined by whether the user lives in the US.
Long story short, AI is a tool that solves problems for companies that actually have those problems at scale. For most companies in the world, product problems aren’t at that scale — and if yours is, then you’ve probably been doing in-house ML for the last ten years anyway. In all other cases, if you see a pitch to integrate some AI-powered recommendation engine or content-analysis system, just think about what +1% would mean for your scale — and ask yourself whether that’s worth another integration and another pointless SaaS subscription.