This publication is broken up into three sections:
TL;DR - For those wanting a quick take
Summary - For those wanting a bit more context and high level points
Article - Main body of work containing full detailed article and explanations that you might want to consume over several readings
TL;DR
One thing we can generally agree on is wanting to build products that customers find valuable. There are a variety of schools of thought and approaches like Balanced Scorecard Metrics, Key Performance Indicators, Measuring What Matters/OKRs, Flow Metrics at a macro-level and, Web, Product, Customer Journey, and Lean Product Metrics at a micro-level.
To operationalise metrics you need to build Insight Analysis into your way of work by developing a process and structure to be able to move from Insight to Action to Impact to enable fast feedback loops. Some things you can do to get you on the path to data-informed organisational and product development:
Create a Logic Model of how your organisation creates, delivers and captures value.
Build a Value Driver Tree so you can see the key variables you can influence to generate the outputs and outcomes that you want. Depending on complexity of your business (e.g., Value Chains, Product Portfolios and Lines, Lines of Business etc.) you may need to develop several Value Driver Trees to approximately describe your organisation.
Build Insight Analysis Process into your way of work to enable Insight to Action to Impact.
"If you do not instrument to track user behaviour, you will not know what is happening with your product or systems. If you do not know what is going on, you cannot possibly be a great operator. You will not know what to focus on to get the outcomes and impact you desire."
Despite metrics being important to measure they are open to abuse and misuse if not developed in balanced way as evidenced by Wells Fargo performance measurement scandal hence the need for counter metrics. A counter metric is something that you measure to ensure that you have not over-optimized your north star metric to the detriment of your customers and your business.
Summary
One thing we can generally agree on is wanting to build products that customers find valuable. So it naturally follows, how can you begin to develop a perspective on knowing what to measure to show value? There are a variety of schools of thought and approaches like Balanced Scorecard Metrics, Key Performance Indicators, Measuring What Matters/OKRs, Flow Metrics at a macro-level and, Web, Product and Customer Journey Metrics, Lean Product Analytics at a micro-level.
We need to use metrics linked to customer behaviour to understand value. A metric is a meaningful measurement taken over a period of time that communicates vital information about a process or activity, leading to fact-based decisions. Metrics are usually specialized by a subject area.
We generally measure things when we want to improve against a baseline level of performance. Performance metrics should be constructed to encourage performance improvement, effectiveness, efficiency and appropriate levels of internal controls.
The challenge today is that we have a lot valuable material that has been developed and shared with respect to metrics like AARRR, HEART + GSM, Customer Goal Attainment etc. however this content does not explicitly show how it connects to a wider organisational perspective.
Building a value framework with a coherent theory or hypotheses about how you generate value is key to making sure that you focus your organisation’s teams’ time, effort and resources on the most valuable opportunities, problems or challenges.
A value driver tree is a good way to showcase your mental model of how your organisation generates value. A value driver tree is a way of visualizing a model of a business in a way that links the value metrics (what management or stakeholders care about) to operational drivers (the things that can be influenced to change the value metric).
Build Insight Analysis Process into your way of work by developing a process and structure to be able to move from Insight to Action to Impact incorporating fast feedback loops.
"If you do not instrument to track user behaviour, you will not know what is happening with your product or systems. If you do not know what is going on, you cannot possibly be a great operator. You will not know what to focus on to get the outcomes and impact you desire."
Despite metrics being important to measure they are open to abuse and misuse if not developed in balanced way as evidenced by Wells Fargo performance measurement scandal hence the need for counter metrics. A counter metric is something that you measure to ensure that you have not over-optimized your north star metric to the detriment of your customers and your business.
Article
Value and Metric Measurement
Question - “What is the one thing that most people in a Product Development function and wider Organisation at large, can agree on?”
Answer - “To build products that customers find valuable.”
So it naturally follows, how can you begin to develop a perspective on knowing what to measure to show value? There are a variety of schools of thought and approaches like Balanced Scorecard Metrics, Key Performance Indicators, Measuring What Matters/OKRs, Flow Metrics at a macro-level and, Web, Product and Customer Journey Metrics, Lean Product Analytics at a micro-level.
The most obvious thing you would think at this point is that everything in your organisation should be measured. This is a mistake that will leave you awash with ‘data, data everywhere and not a single insight to action’. Rather your approach should aim to develop a mental model of how your organisation generates value.
But before we get lost in the maze of measurement and metrics we should go back to some foundational basics. For instance since we can agree on building products that customers find valuable then it means we need to first agree on what constitutes value?
In my time developing product this is probably one of the most important and foundational issues to align a product development team on. Developing a shared understanding on what value your product or service provides is an important leverage point that is context dependent i.e., the metrics you would develop for a B2C media product like Netflix are different from the metrics used to evaluate specialised B2B enterprise software from like a Cambridge Mobile Telematics product targeted at P&C Insurance companies.
To identify what constitutes value we need to identify the key actions or jobs to be done by our customers or end users.
Question - “What specific ‘jobs to be done’ are my users hoping to do with my product? And what is my product really about?”
To answer the above question there are three main issues to be addressed:
Product Niche - What market are we in and what specific jobs to be done are we helping our clients to achieve?
Product Value - What value do we help to create for our customers and end-users and how as an organisation do we partake in the value creation process?
Product Value Exchange Mechanism - What mechanics underpin the value exchange e.g., transactions, subscriptions, in-app purchases, attention, advertising or data monetisation etc.?
If you look at the most successful products today, they have a few things in common? They either make our lives easier, cheaper, more convenient or, play a social status signifying role to get what we want (e.g., think Amazon Prime or Checkers sixty60 for same day delivery) and they will have a value moment which is an event, an action, or a series of events and actions that represent the moment that a user found value in your product.
The key to having a solid perspective on customer or end user value is to be able to identify the specific behaviors that indicate that people are getting value from your product. These behavioural markers are key to knowing if you are creating, and delivering value.
By knowing when and how your users get value from your product you can optimise the path to that critical value moment for more users, which means more people get value and are willing to keep coming back, and perhaps if you are really good you can capture a meaningful portion of the value you create through monetisation of key behavioural actions.
Creating a Value and Metrics Framework
Question - “So what underlying concepts do we need to measure behavioral markers or system changes?”
Metrics:
A metric is a meaningful measurement taken over a period of time that communicates vital information about a process or activity, leading to fact-based decisions. Metrics are usually specialized by a subject area.
We generally measure things when we want to improve against a baseline level of performance. Performance metrics should be constructed to encourage performance improvement, effectiveness, efficiency and appropriate levels of internal controls.
Some Characteristics of a Good Metric:
Defines an event that is simple, understandable, logical and repeatable
Allows a stakeholder to move from Insight to Action to Impact
Meaningful to a stakeholder (e.g., customer, end-user, member, donor, etc.)
Collection of data is economical and cost effective
Shows how organizational goals and objectives are being met
Leading and Lagging Indicator Metrics:
Leading Indicator Metrics aka Input Metrics in the language of Amazon: A leading indicator is any measurable or observable variable of interest that predicts a change or movement in another data series, process, trend, or other phenomenon of interest before it occurs. Leading indicators in organistions can be captured via operational metrics showcasing live performance that have a link eventually to business outcomes. Leading indicator metrics are predictive of Lagging indicators.
Lagging Indicator Metrics aka Output Metrics in the language of Amazon: A lagging indicator is an observable or measurable factor that changes sometime after a business variable with which it is correlated changes. Lagging indicators are typically captured as financial, customer or overall business metrics.
Additional Considerations on Metrics:
When it comes to metrics there are some pitfalls to using data and metrics. Brian Balfour of Reforge captures some of the most pertinent issues with respect to use of data and metrics in the below quote.
The most common pitfall is relying either too much on intuition or too much on data. You have to start with a qualitative hypothesis of what the value moment is, based on first principles of your product’s problem, target audience, differentiation, and other elements. You then need to validate that hypothesis with certain analyses. Teams that let the data drive the definition are letting the data dictate the strategy, when the data should be used to measure whether or not you are achieving the strategy. Those two things are fundamentally different.
The best leaders understand that a single metric can’t capture the entire picture of their business, but that one metric at any given time can be the most important. It is a balance.
Most products should have four high-level metrics: an acquisition metric, retention metric, engagement metric, and monetization metric. The four of these act as a system where one can influence the other. They each capture a different dimension of what is happening with a product. But trying to improve all metrics at once typically isn’t possible until you get to a much later stage and are able to dedicate teams to each area. Instead, leaders need to understand which one is most important at any given moment and make sure the teams understand that and are properly resourced to make meaningful progress.” Brian Balfour, CEO Reforge talking about Metrics in a Mixpanel Report on product analytics
Getting Logical on Value and Metric Measurement
Question - How do we systematically build a value measurement framework that is not context specific since we can generally agree that measuring and tracking value metrics for customers, end users and the business is important?
The challenge today is that we have a lot valuable material that has been developed and shared with respect to metrics like AARRR, HEART + GSM, customer goal attainment etc however this content does not explicitly show how it connects to a wider organisational perspective.
Building a value framework with a coherent theory or hypotheses about how you generate value is key to making sure that you focus your organisation’s teams’ time, effort and resources on the most valuable opportunities, problems or challenges.
After working on product development in an Enterprise space one thing I have realised is that the metric frameworks pushed out in the wild are not always applicable within all contexts like an Enterprise software technology company setting. Usually you have to think really hard about how the products you build generate value for your client and link to key customer or end-user actions/behaviors (i.e., leading indicators) which are predictive of business outcomes (i.e., lagging indicators).
I was fortunate to come across the Logic Model Framework which was documented and popularised by the Kellog Foundation.
Logic Model Framework
As an aside point, the Logic Model with a bit of lateral thinking can be linked to the Business Model canvas. This framework is flexible enough to allow you to generate context specific leading indicator metrics that are actionable and lagging indicator metrics to highlight performance.
The beauty of this framework is that it tries to get you to think through causality chains and how you can link the resources your organisation has access to, to how your business activities/processes generate outputs and the eventual outcomes and business impact that you generate.
Ideally you should work backwards from the change (i.e., Impact) you wish to create to the resources (i.e., Inputs) you have that can help to unlock the impact you wish to make:
Impact - Big picture change is only known after the fact, and is not actionable per say (e.g. lagging indicator metrics). Impact is the fundamental intended or unintended change occurring in organizations, communities or eco-systems as a result of your organisations activities within a 5 to 10 year period. This is generally when you can see how well you have performed in executing your strategy and meeting your organisational objectives and goals.
Outcomes - Outcomes are lagging indicators that represent a change in human behaviour that creates value. Outcomes are specific changes in a customer or end-users behavior, knowledge, skills, status and level of functioning. Outcomes can be broken out into short-term outcomes that should be attainable over a 12 to 24 month period while longer-term outcomes should be achievable within a 2 to 5 year time frame. The logical progression from short-term to long-term outcomes should be reflected in impact manifesting within a 5 to 10 year period.
Outputs - Things an organisation does in order to create an outcome. Direct products produced from organisational activities and may include types, levels and targets of services to be delivered as well.
Activities - Key activities required to generate an output using some key resources as an input. Activities are the processes, tools, events, technology, and actions that are an intentional part of the program implementation. These interventions are used to bring about the intended program changes or results.
Resources - Inputs used in a upstream process to generate outputs. Resources include the human, financial, organizational, and community resources an organisation has available to direct toward doing the work. Sometimes this component is referred to as Inputs.
Practical and Pragmatic Approaches to Value and Metric Development
Question - So how do I begin thinking through developing a value creation, delivery and capture framework?
A conceptual framework is needed for the performance measurement and management system. Every organization needs a clear and cohesive performance measurement framework that is understood by all levels of the organization and that supports objectives and the collection of results.
Every organisation will need to develop a custom approach to metrics.
Performance measurement needs to involve both senior management who understand the long-range goals of the organization, and front line staff who know the kinds of measures needed for thoughtful decision making. Although there is no “perfect set” of measures, organizations should experiment with a “starter” set of metrics and evolve these measures over time.
Some parts of AARRR-pirate model could be applicable in your context (i.e., Acquisition and Activation but each of these points will have their own meaning depending upon the nature of your product and certain customer milestones) but it is useful to tie metrics to specific value levers of your product (if any) e.g. a submit claim journey and its associated success metrics and how it ties to business outcomes e.g. cost per claim, claim processing time, claim error rates for an Insurer etc.
The biggest challenges in measurement are not in the collection and analysis of data, but in interpretation of results and making decisions based on the information. Education is an essential part of all measurement.
Knowing what to measure, how to measure, how to use the measurement to inform action and how to link these measures to other knowledge of organizational performance are critical to the successful use of performance measurement.
To successfully make use of metrics you need to provide training to teams on this and set up a cadence-based process for reporting, reviewing and improving i.e., Weekly, Monthly or Quarterly Business Reviews.
Effective communication with employees, process owners, customers and stakeholders is vital to the successful development and deployment of performance measurement systems.
It is the customers and stakeholders of an organization, whether public or private, who will ultimately judge how well or not if you have achieved your goals and objectives. And it is those within the organization entrusted with and expected to achieve performance goals and targets who must clearly understand how success is defined and what their role is in achieving that success.
Both organization outsiders and insiders need to be part of the development and deployment of performance measurement systems. Accountability for results must be clearly assigned and well-understood. High-performance organizations clearly identify what it takes to determine success and make sure that all managers and employees understand what they are responsible for in achieving organizational goals.
Performance measurement systems must provide intelligence for decision makers, not just compile data. Performance measures should be limited to those that relate to strategic organizational goals and objectives, and that provide timely, relevant and concise information for use by decision makers at all levels to assess progress toward achieving predetermined goals.
Now some may ask does this not close the space down for exploratory analysis? Not really it just requires that those proposing to consider metrics or analyses beyond the value creation framework need to have sound and crisp logic showing how the things they wish to analyse link back to how the organisation creates value.
Question - How could the metrics in your value framework be organised that tell the facts of the business and structured into a compelling story-line?
A value driver tree is a good way to showcase your mental model of how your organisation generates value. A value driver tree is a way of visualizing a model of a business in a way that links the value metrics (what management or stakeholders care about) to operational drivers (the things that can be influenced to change the value metric). In this respect a value driver tree is the visual representation of a mathematical model of a business (or a portion thereof).
The value driver tree is useful because:
It is visually appealing and engaging
It shows how different areas of responsibility (e.g. engineering and operations) link together and affect the value metric
Additional benefit of modeling value through value driver trees is that they can be used to model specific parts of value chain and not just an organisation as a whole, always remember to keep the big picture in mind even when working on a narrow area
Question - Some of you might be thinking to yourselves, the above makes sense in a profit making context but does the same apply in a non-profit or public sector setting?
I would say that the above is a valid question however even in both the non-profit and public sectors we need to be able to show the impact we create from the resources (i.e., tax revenues, grants or donations) that have been entrusted to us by the public at large.
Table Comparing Points of View on Metrics Across Private and Public Sectors
Question - So you may be thinking I get where you are coming from Tshepo, so what should I do to develop insights that move me to action and eventually generate impact?
I would suggest the following:
Create a Logic Model of how your organisation creates, delivers and captures value.
Build a Value Driver Tree so you can see the key variables you can influence to generate the outputs and outcomes that you want. There is a reason that the best management consulting firms emphasise the use of issue trees in the early tenure of a consultant. Depending on context and complexity of your business (e.g., Value Chains, Product Portfolios and Lines, Lines of Business etc.) you may need to develop several Value Driver Trees to describe your organisation.
Build Insight Analysis Process into your way of work by developing a process and structure to be able to move from Insight to Action to Impact incorporating fast feedback loops like an OODA loop.
Closing Remarks
NB. Build to Measure ( Golden Rule – Don’t ship before you have defined your value driver tree and your associated metrics)
You need to make metrics definition as part of your “definition of ready / done” (i.e, a criteria that you can assess user stories against in order to determine whether they can be considered defined and ready to be consumed by Customers and End-users).
You should not define your key metrics after a product has launched as this is not a recipe for success.
You should build metrics testing into your QA process to make sure the data you collect is accurate before trying to quantify the unknown. This is a great way to ensure everything you do is measurable where possible.
You should not be releasing into production until you have identified the metrics that matter.
If there is one thing that you remember to do from this article:
"If you do not instrument to track user behaviour, you will not know what is happening with your product or systems. If you do not know what is going on, you cannot possibly be a great operator. You will not know what to focus on to get the outcomes and impact you desire."
PS.
If you are moved please leave feedback so I can improve the publication and topic coverage.
Please share this article to your wider community, colleagues, friends or family that could benefit from this content.
Post Post Script
Sample Ideas for Metrics -
Obviously depending on your context you will need to develop metric definitions that are meaningful to your specific business and capture some of the secret sauce that creates differentiation from your competitors.
Business Metrics
“What are the metrics that matter to Stakeholders?”
Financial metrics will generally always be the most important KPIs. Note, the product business model will also influence the definition of your key metrics:
Average Revenue per User (ARPU) or Customer:
Cash Conversion Cycle
Average Annual Contract Value
Gross Margins
Number of new users: weekly, monthly, quarterly, yearly
Active users: daily, weekly, monthly active users
Churn or Retention Rates
Average session time
Engagement Metrics
“Engagement metrics help you solve the “Why”
We need to understand our users and how they are using our product. This is an invaluable tool throughout the product development process. User engagement metrics are important to see how the users interact with your product.
The whole point of these metrics is to learn as you go, to reinvest this information to make better product decisions down the road.
If all we know is whether our investment was successful, but we don’t know why, then we don’t know what our next investment should be.
Key engagement metrics for product features:
Metrics that help us understand how customers are interacting with the product and identify opportunities to grow
How often is each feature being used?
Which feature sets tend to be used by the same kinds of people?
What features are our most engaged customers using frequently?
Where do users get stuck and abandon the product?
How long are they spending on each feature?
Who abandons and who keeps using it?
Are there any seasonal trends that can be used for predicting future revenue and usage?
Potential Failure Modes
Compensation, rewards and recognition should be linked to performance measurements however this linkage should be positive not punitive. Such a linkage sends a clear and unambiguous message to the organization as to what’s important.
Despite metrics being important to measure they are open to abuse and misuse if not developed in balanced way as evidenced by Wells Fargo and performance measurement scandal hence the need for counter metrics. A counter metric is something that you measure to ensure that you haven’t over-optimized your north star metric to the detriment of your customers and your business.
#Value #Metrics #ProductAnalytics #LeadingIndicators #LaggingIndicators #MentalModels #LogicModel
Resources
AARRR Metrics
Amplitude - North Star Playbook
Flow Framework of Metrics
Google HEART and GSM Framework
Article on frameworks for measurement by UXDesign
Lean Product Analytics Book by Alistair Croll and Benjamin Yoskovitz
Dan Olsen speaking about Lean Product Analytics