Why talking to your users can be downright dangerous!
The product manager's #1 task can easily lead them astray!
If you’ve read anything about product management, you know that the One Thing™️ you need to always, always do is talk to your users. While product management is complex, involves a lot of tasks, and requires a multi-disciplinary skill-set, if there is one immutable thing that you need to constantly do is stay in touch with your users. You need to fully understand their context and constraints, and communicate that back.
But what if I told you that speaking to users is fraught with danger, and can lead you down the wrong path?
Profiles of Users
Once you release a product to the market and get some traction, users fall into a normal distribution. There are the users who just started, who are still learning your product and asking lots of questions.
Then there is the majority of users, who have achieved proficiency, and know what to do to get the results they need. Lastly, there is a small number of users who spend the extra effort to become experts in your product. They are truly invested — both in the efforts they made, but also in their emotional connection to the product.
So what’s the problem?
The issue is that you hear a lot from new users (asking questions on how to get things done), and from experts (opinionated on best practices and what new things your product could do for them).
But you don’t hear a lot from the majority of people, who know how to do their tasks (and get the value) with your products. They have learnt to accept or work around any limitations, and are more concerned with just getting along with their day.
This is especially prevalent with B2B/Enterprise software. Users are so used to crappy, poorly-designed experiences that they just don’t think there’s a point in fighting this. In extreme cases, users start thinking that there is something wrong with them for not understanding the software and knowing how to work it properly. I mean, look a all those wizards who can make the product dance!
So what can you do? If the majority of the users are potentially facing challenges you aren’t hearing about, and yet those who do come forward to speak with you have biases to the extreme? How do you ensure you build for the 80% of mass market?
Employ user proficiency to your advantage
Once you realise the dangers of accepting all users’ feedback equally, their stance becomes obvious. It’s easy to categorise users experience level and then understand the context they are speaking from.
No need to get overly detailed, a simple self-declaration of beginner / proficient / expert is usually enough. If you have enough statistically significant sampling and want to collect how long they’ve been using the product to chart this and draw future conclusions, all power to you.
But speaking with users is inherently qualitative, a more interesting observation is what you can learn from each segment.
Beginners are those who got their product in order to solve their problems. They aren’t always sure how they could do that, so they ask questions. The obvious aspect to improve is the ob-boarding flow. A good on-boarding experience, delivering the quickest time-to-value, is the best leading indicator of long-term retention across industries. But there’s a deeper opportunity here that’s worth pursuing. talking to users who have their problem on their mind and are still somewhat untainted by preconceptions of solutions gives you a great opportunity for some problem-space discovery. It’s the perfect opportunity for you to dig in to the jobs they are trying to accomplish, and understand how your product fits that market.
Advanced users will often come to you with great ideas about new problems that your product could solve. If you just add that we could do this, or make that workflow so much more efficient, or create that super-sticky super-tool, or whatever. And it’s possibly true… for them. It’s your job to understand whether what they are suggesting aligns with your company’s strategic direction (do you want to compete in that market?), whether there is a sufficient market behind, a sufficient number of people with burning need to make the effort of building the solution into a viable business.
Lastly, the ‘silent’ majority offers a mix of aspects you could learn about. They do require more effort to reach out to and interview, and perhaps more care in the questions you ask, but it’s worth the time. These are people are using your product and deriving value, but have gotten comfortable with it and are perhaps blind to some faults. You can explore both the problem space by asking more open-ended questions and the solution space of tweaking and improving the experience. Continually polishing the experience, removing unused features (or hiding them so only advanced users can access them) and simplifying the overly complicated workflows will increase user satisfaction and retention. Understanding the problems the majority of users face will also help you double down on where the biggest market is.
In Conclusion
As a PM, you absolutely must do that One Job™️ of talking to constantly customers. You just need to be more conscious of their background and context, and what you can learn from them. This is essentially a qualitative exercise, which teaches you a lot about the problems they are using your product to solve. You should then use the combination of their feedback and context to measure it against your the yard-stick of your product strategy, and those opportunities that align can (and should) be quantitatively validated for broad-market applicability.
Bonus!
A user’s level of proficiency isn’t the only thing that shifts over time. I’ve written before on the Kano Model, and how the perception of what’s shiny vs what’s just table stakes shifts over time.
But it turns out the whole user base, indeed the whole market, shifts over time. In Crossing the Chasm Geoffrey Moore describes the reason many companies fail. They see the interest from the innovators and early adopters — who are fundamentally excited by the technology and are more forgiving of faults and missing functionality — and then fall into a chasm between them and the early majority. If we think back on the Kano model, the early majority are far more concerned with the breadth of the solution, that in comparison to the competition (including the status quo) it offers all the table stakes and the functional features they need.
Furthermore, once you get over the early majority into late-majority and laggard territory, the whole market shifts perception. Products and features are commoditised, with lower differentiation. As the market matures it attracts more competitors, and amongst users there’s a bigger drive to get a smoother experience of the basics. Companies start hitting the Innovator’s Dilemma of risking losing market share if they disrupt themselves with new innovation.
So not only do you need to understand the context of the individual user you are currently interviewing, but you also need to understand where your company is in relation to the whole market segment and which user profile is it attracting. A new user of a startup’s early adopter is more akin to the expert or a later stage company’s product in concerns and mentality.
Talking to users is the One Job you have to do as a product manager, but you need to be keenly aware of the context both you and the user operate in to filter any biases. You need to understand what you can learn from each user, how their feedback might be coloured, and what would be the potential next steps in acting upon it.
This was originally part of the Product Discovery talk for Leading the Product (a recording of which is available behind their Season’s Pass), but as user interviews is such a fundamental activity I thought it warranted a long-form post as well.
I’d love to hear your thoughts on this!