Estimating the impact that product work will have is always a mixture of art and science, and never leads to precise answers. However, it’s an invaluable tool at all levels of a product organization that helps you get a handle on some of the most fundamental decisions you need to make as a product manager - which problems are worth solving, and how to solve them.
“All models are wrong, but some are useful.”
– George Box, statistician
This article is about how you estimate the impact you’ll have as a product team, and how to use these estimates, both to make better decisions for your users, but also better decisions for the business. As well as talking about why modeling impact is important, we’ll talk through the best practices of doing it, and share a template to get you started.
What is an impact model?
An impact model is a spreadsheet that helps you estimate the impact a feature or a theme of work will have, as well as understand how important different drivers are to creating impact.
What it does:
- Help you understand how metrics are related
- Help you think through how user behavior drives business impact
- Help you estimate the size of impact product work could have
- Help you forecast the financial impact that product work will result in
- Help you compare the impact you can expect from different pieces of work (big things vs small things)
- Prevent you making avoidable mistakes and damaging your credibility by adding work to the roadmap which is low impact
What it does NOT:
- Give you certainty over whether features will be successful
- Predict precisely the impact that new product features will have
- Forecast precisely the financial impact that new features will have on the business
When you create a model, you are creating a rough simulation of your product and user behavior so you can test your hypothesis for building a feature. As you change things, you can see how connected metrics are affected, which can help you evaluate whether your hypothesis is credible, and understand what might happen in the future.
One of the most common applications for modeling features is to help you assess how large an impact they will have. This will help you answer questions like:
- Is this feature likely to have a big impact, or is it more of a small optimization?
- When I compare this to the cost of a sprint, does developing this feature seem worth it?
- Do I have enough high impact features in my roadmap?
If you have a lot of small things in your roadmap which don’t move the needle very much then you need to revisit what the team is working on. Product teams are expensive, and teams have a responsibility to deliver a return on the investment the company is making in them.
Want to go deeper on strategy? Check out our 7 week course: Product Strategy
Why model impact?
"At Captify all PMs have to estimate potential revenue impact for each epic as part of our prioritisation process."
– Amelia Waddington, CPO Captify
Modeling the impact you will have has several benefits. It will help you to:
- Prioritize work - having some sort of estimate for the impact work will have is an important input into prioritization. There’s always a large amount of uncertainty in estimating what will happen when new features launch, but creating an impact model prevents obvious mistakes, like working on features that have no clear mechanism for changing key metrics, or on features that are only used by a small proportion of users.
- Achieve targets - most businesses set teams targets on a regular basis (e.g. quarterly OKRs), so they can measure how well they are doing, and to motivate them. If these targets are unrealistic, then people ignore them. The only way to get realistic targets is to have some sort of idea of what will happen when a team works on a problem for a given amount of time. Understanding the size of the levers and what is achievable in a given time frame is key to setting and achieving business targets.
- Forecast the future - CEOs and CFOs need to forecast the financial performance of the business into the future so they know if they need to raise money, cut costs, or can invest more aggressively. The better that you can understand how your product teams will change business performance, the more accurate your forecasts will be.
- Justify investment - teams often want additional people or investment to support their plans. Having a model of the impact that this kind of investment will lead to makes this much easier to justify and approve.
Managing stakeholder expectations
When building impact models, it’s important that everyone understands the amount of uncertainty involved. Stakeholders will undoubtedly get frustrated if they’ve been led to believe a feature will definitely deliver results, but this doesn’t happen when the feature goes live.
Actual impact can only be determined through A/B testing or other detailed analysis when a feature goes live. Everything pre-launch is a prediction which can be more or less accurate. How accurate your model is depends on the quality of data that you can put into it. If you’re building a 0 -> 1 new product, then it’s very difficult to assess what the impact could be, as you have no benchmarks. But if you’re doing further iterations to a feature that you’ve been working on for some time, then you should be able to be fairly accurate.
However, this lack of certainty doesn’t make impact modeling worthless. Far from it. Having some idea of how much impact a feature will have is much better than having no idea. Presenting senior leaders with roadmaps that have no thought to the impact they will create will severely undermine your credibility, so this is a skill that you need to develop. Senior leaders are used to navigating uncertainty. Have a thorough conversation with them about what they should and should not read into your model.
