This publication is broken up into three sections:
TL;DR - For those wanting a quick take
Summary - For those wanting a bit more context and high level points
Article - Main body of work containing fully detailed article and explanations that you might want to consume over several readings
TL;DR
This article explores the evolution of data analytics in business from basic Web Analytics to Digital Experience Analytics (DEA).
My prediction for the future of analytics platforms is that the capabilities found in Web Analytics, Product Analytics, Customer Journey Analytics and Digital Experience Analytics platforms will converge into a single platform.
The foundational capabilities required in this Next Gen Platform are business event definition, robust data management, data integration connectors with business systems, real-time processing, data visualization and reporting, privacy and security, and scalability and performance.
Furthermore, I share a roadmap of evolving features in this Next Gen Analytics Platform: V0 focuses on essential data management and infrastructure, like data ingestion, processing, integration, privacy, governance, and audit. V1 incorporates descriptive, diagnostic, predictive, and prescriptive analytics for comprehensive data analysis. V2 adds automated workflows, machine learning operations (MLOps), collaboration, and edge analytics for IoT data processing. V3 introduces AI explainability, AutoML, data virtualization, embedded analytics, natural language processing, advanced visualization, and a marketplace for analytics apps. I also tease that the V4 version could leverage AI to develop unprecedented features.
The importance of balancing added features with usability and simplicity is also stressed. I highlight the role of orchestration capabilities for decisioning, experimentation, and actioning in testing new ideas and providing personalized user interactions.
Summary
The article below discusses the evolution of data analytics for businesses, moving from Web Analytics to Product Analytics, Customer Journey Analytics (CJA), and finally to Digital Experience Analytics (DEA) and how the forementioned approaches are converging into similiar feature sets.
Initially, Web Analytics was a key tool for understanding website performance and basic user behavior. It evolved from simple hit counters to advanced tools like Google Analytics, but faced limitations in tracking individual user behavior.
This led to the rise of Product Analytics, which focuses on analyzing user interactions with digital products to enhance features and improve user experience. It moves away from the traditional page view-centric approach to a user and behavior-centric one, while offering benefits like advanced segmentation capabilities and the ability to discover "unknown unknowns" in product and service performance.
The next evolution was the focus on CJA as a part of Customer Journey Management (CJM). This broadened the view from specific touchpoints to the entire customer experience, integrating data from both online and offline channels.
The current stage is DEA, which collects data on user behavior, engagement, preferences, and interactions across digital channels, aiming to optimize the customer journey. Features of DEA include session replay, heatmaps, form analytics, journey discovery, and segmentation.
My personal view is that these different forms of analytics will ultimately converge into a single platform, either offering a comprehensive set of analytic tools or a modular system where the best analysis tools can be integrated into one analytics workbench area.
Article
My first article in this series spent quite a bit of time explaining how you can develop actionable insights through analytics in order to drive business value and impact. One of the major points I emphasise is the need to integrate data from various touchpoints, which are the various devices and channels through which customers or key stakeholders interact with an organization.
In a follow-up article, I explored Web Analytics which was one of the first internet era digital technologies used to understand website performance and user behaviour. The historical development of web analytics evolved from hit counters and server log data to manual tagging and the evolution of tools like Google Analytics. The limitations of traditional web analytics tools led to the need for more sophisticated approaches to track user behaviour and data analysis like Product Analytics.
Product Analytics emerged as the successor of Web Analytics and emphasised the importance of shifting from a page view-centric approach to a user and behaviour-centric approach across smartphone and web applications. Despite the rise of Product Analytics, many businesses still rely on aggregate metrics from legacy Web Analytic tools, such as pageviews and bounce rate, which provide limited visibility into individual user behaviour and interactions across different platforms. Product Analytics is a practice that focuses on gathering, analysing, and interpreting data about the usage of digital products to improve user experience and enhance product features. The benefits of Product Analytics include advanced segmentation capabilities made possible through dimensional modelling, and the ability to discover "unknown unknowns" in product and service performance.
The next stage in the evolution of analytics has been the focus on Customer Journey Analytics (CJA) which forms a part of Customer Journey Management (CJM) and is important in helping you to understand and improve the whole customer experience (CX) and not just interactions at specific touchpoints. CJM is needed to handle the complexity of managing interactions between customer segments, devices, channels, and capabilities across the entire customer lifetime. CJA goes beyond web and product analytics by integrating operational data from online and offline channels, think about how you could integrate data across touchpoints accessed through USSD, branch offices, remote kiosks, alongside web and smartphone app usage etc. The components of CJM, include journey mapping, journey visualization or discovery, and journey orchestration. Besides analysing end-user usage or behaviour, it is also critical to see what the actual digital experience was like.
Digital Experience Analytics (DEA) arose as a crucial tool for understanding, measuring, and optimizing user interactions across digital channels. There is some overlap at present between DEA, Product Analytics and CJA. DEA enables businesses to gather data on user behaviour, engagement, preferences, and interactions to improve the customer journey. Major DEA feature sets include session replay, heatmaps, form analytics, journey discovery, and segmentation.
My personal view is that the various forms of analytics are going to converge into a single platform capable of either providing out the box the major analytics needs or enabling a modular plug and play option where the best of breed analysis tools can be combined into a single analytics workbench area.
The platform required to support this convergence will require key foundational capabilities.
Foundational capabilities required for a next generation analytics platform
There are some foundational capabilities that are required to enable a data-enabled and, user behaviour-informed organisation.
Foundational capabilities:
Built-in Business Event Definition -
Business Event Identification: Business event definition requires identifying and defining key business events and customer interactions that are relevant to an analysis, such as website visits, task completion, product purchases, customer support interactions, and social media interactions.
Business event definition is required so as to create a shared context and language on what certain events or actions and associated attributes mean. In some cases there might be internationally defined standards like ISO or ICD-10 standards that are applied in your industry. In others, these things may not be universally defined or applicable like what actions contribute to someone being categorised as a DAU, WAU or MAU for a social media company. Your categorisation framework will depend on how your organisation wants to define the goals of your products or services.
Robust Data Management -
Data Taxonomy: Developing a well-defined and structured classification system for organizing and categorizing data events, elements, attributes, and metrics to facilitate consistent data management and analysis.
Customer Identity Resolution: Resolving and matching customer identities across different touchpoints to create a cohesive customer profile.
Data Integration: Only once key business events and processes are identified does it make sense for Data Integration which requires integrating data from various sources, such as websites, mobile apps, business applications, CRM systems, and social media, to create a unified view of customer interactions. Customer Data Platforms (CDPs) can play a role here but not all relevant business event data is captured in CDPs for example.
Data Governance: Data governance is required to ensure data quality, business value and accessibility to support secure access to accurate and reliable analytics.
To enforce data quality it is critical to have a defined data management process, data operations lifecycle management and corporate governance. This is key especially if an organisation needs to have trust in the data being produced and used to make critical decisions or changes to operations. Without sound data management practices it will be increasingly difficult to build AI algorithms that can take advantage of your unique data insights.
Data Integration with Customer Interfaces and Operational Systems -
Integration with Business Technology System Platforms: Integrating omnichannel data with back-end business operations platforms to enable personalized and targeted customer insight activations across marketing, sales, support and servicing etc.
AI-Powered Personalization: Leveraging artificial intelligence and machine learning algorithms to deliver personalized experiences and recommendations to customers across multiple channels when they want on their preferred channels.
Integration capabilities are key to ensuring that you can pull together data from disparate systems and create unified and integrated data sets that can tie unique customer identities to various interactions across touchpoints. We have all been on calls with contact centre agents where even though they are still in the same functional or organisational unit, context gets lost when you are transferred to another agent for whatever reason.
Real-Time Processing -
Auto Capture: Enable event auto capture with built-in safe-guards to exclude sensitive data from being captured in a programmatic way.
Real-Time Data Processing: Processing data in real-time as it is generated to enable immediate insights and actions. Technologies such as Kafka will play an important role in enabling Data in Motion which can be processed in individual units of data as well as in batch. If you think about it from a Lean perspective, streaming can allow you to process data in an idealised unit basis but other issues creep in around consistency, reliability etc.
Advanced Analytics: Utilizing sophisticated analytical techniques, such as machine learning and predictive modelling, to derive meaningful insights and make real-time inferences from complex data sets.
Near real-time or real-time views could provide business critical functionality. Consider things like eCommerce or payments fraud as examples requiring low latency between fraud being committed and transactions as being flagged as being suspicious or fraudulent.
Visualization and Reporting -
Visualization and Reporting: Presenting data and insights in visually appealing and easily understandable formats through charts, graphs, and interactive dashboards to facilitate data-informed decision-making.
It is also critical to be able to tell a story making use of visual aids. In many cases it is a lot easier to communicate an insight using a visual graph or infographic instead of pure text. Hacking the human mind is maybe another line to explore concerning getting buy-in for data and analytics.
Privacy and Security -
Privacy and Security: Ensuring the protection and compliance of customer data by implementing robust privacy measures, data encryption, and adherence to relevant regulations and policies.
Data Retention: Enable data retention policies and governance that are compliant. Prove compliance with regulatory record keeping requirements through programmable data retention.
In the current context in which we operate in. Privacy by design will most likely become the default way to manage how sensitive data needs to be handled. Various countries have varying data and privacy requirements. Any analytics platform will need to have a robust architecture and ability to programmatically handle differing country or regional requirements.
Scalability and Performance -
Scalability and Performance: Building a scalable infrastructure capable of handling large volumes of data and providing efficient processing to support the growing demands of omnichannel analytics.
The platform required to handle data from across a range of sources, formats and processing times will need be highly performant and resilient against faults or failures. A large chuck of the technologies to enable this already exist, the question is can someone bring these capabilities together in a cost effective way?
Major value-adding capabilities for analytics
From my perspective, it is important to craft categories that to a large extent are MECE – Mutually Exclusive Collectively Exhaustive. Thanks to Barbara Minto of Pyramid Principle fame for crafting such a useful logic synthesis tool.
One way to think through the analytics landscape is through the four following areas:
Descriptive Analytics – Involves interpretation and analysis of past events or historical data. It is useful for understanding questions related to what has just happened?
Diagnostic Analytics – Diagnostic analytics goes a step further than descriptive analytics by providing insights into why something happened. It’s useful for identifying what is the specific root-cause of something?
Predictive Analytics – Predictive analytics employs forecasting techniques and statistical models to understand the future. It’s useful for asking questions related to what could happen given a set of historical data or information? Often the techniques involved include regression analysis, multivariate statistics, pattern matching, predictive modelling, and forecasting.
Prescriptive Analytics – Prescriptive analytics goes beyond predicting future outcomes by suggesting actions to benefit from predictions. It uses optimization and simulation techniques to advise on possible outcomes. It is useful for giving direction or recommendations on specific courses of action.
In the next few sections I will group a non-exhaustive list some analyses that can be grouped into the above categories. This approach I believe can cover a significant portion of your analytics needs, e.g. 99.99% ;).
Analytics Use Cases
Descriptive Analytics (i.e., descriptive statistics or after the fact analysis)
Segmentation Analytics
Segmentation according to business defined dimensions ala Ralph Kimball dimensional modelling
Cohort analysis based on products, dates or any other relevant dimensions
Marketing Analytics
Awareness metrics
Acquisition metrics
Activation, Adoption, Conversion and Engagement metrics
Product Analytics
Feature usage metrics
Product key performance indicators
Operational Analytics
Flow metrics
Cost to serve metrics
Application performance monitoring metrics
Business Key Performance Indicators
Revenue
Cost of Sales (CoS) or Cost of Goods (CoGs)
Retention rates
Referral rates
Customer Acquisition Costs - CAC
Customer Lifetime Value - CLTV
Payback periods
Examples:
Retail companies use descriptive analytics to analyse customer purchase histories and understand customer behaviour.
In healthcare, descriptive analytics might be used to understand trends with in-patient admissions.
Telecommunications companies use descriptive analytics to understand call patterns and trends.
Financial institutions use descriptive analytics to understand past financial performance and trends.
Diagnostic Analytics (i.e., root-cause analysis)
Workflow management to build robust understanding of business processes and workflows
Unpacking contribution analysis
Feature management analysis
Crash analytics
Failure rates
Success rates
Struggle and error analysis
Error % rates
Task % completion rates
Session replays
Interaction heatmaps
User behaviour and timeline
Customer journey analytics aka user journey flows (e.g., path discovery and analysis)
Voice of Customer Tools (i.e., Qualitative Feedback)
Examples:
An e-commerce company uses diagnostic analytics to understand why sales dipped in a particular quarter.
A manufacturing company may use diagnostic analytics to understand the reasons for an increase in product defects.
A healthcare provider might use diagnostic analytics to determine why certain patient groups are readmitted more frequently.
A telecom company could use diagnostic analytics to determine why certain customers are experiencing dropped calls or other service issues.
Predictive Analytics (i.e., forecasting, probability of default, churn risk etc.)
Forecasting
Propensity modelling
Anomaly detection
Churn predictions
Fraud detection
Examples:
Credit card companies use predictive analytics to determine the probability of default for individual customers.
Retailers use predictive analytics to forecast sales and manage inventory levels.
Insurance companies use predictive analytics to calculate risk and determine premium rates.
Marketing companies use predictive analytics to forecast the success of campaigns or promotions.
Prescriptive Analytics (i.e., recommendations or suggestions or nudges etc.)
Recommender Systems
Next Best Actions for Personalised Offer Decisioning
Intelligent Alerts
Examples:
Airlines use prescriptive analytics to determine optimal ticket pricing.
A manufacturing company uses prescriptive analytics to optimize supply chain logistics.
Energy companies use prescriptive analytics to manage power grid distribution.
Retailers use prescriptive analytics to determine the best locations for new stores based on demographic data, competitor locations, and predicted future demand.
These are the four types of analytics. They represent a continuum from looking at the past to predicting and influencing the future. They help businesses understand their past performance, why things happened, what might happen next, and what to do to meet their business goals.
Having a systematic approach to incorporating these analyses into your way of work is also critical in order to extract value of from analytics.
Some form of orchestration for decisioning, experimentation and actioning workflow management is critical.
Orchestration for Decisioning, Experimentation and Actioning (O-DEA) –
Experimentation: This is about running controlled experiments to test new ideas or tactics. Features could include:
Tools for designing and running experiments (like A/B tests or multivariate tests).
Statistical analysis tools to interpret the results of experiments.
Reporting and visualization tools to share experiment results.
Orchestration: This is about coordinating various processes or systems in an automated and efficient way. Features could include:
Workflow management tools to automate and coordinate tasks.
Integration tools to connect with other systems or processes.
Monitoring and alerting tools to keep track of orchestrated processes.
Actioning: This is all about interacting with users/customers in a personalized and effective manner. Features could include:
Segmentation tools to target specific user groups.
Personalized content or recommendations.
A/B testing tools for optimizing engagement strategies.
Examples
Testing and experimentation framework
User lifecycle engagement
Audience engagement, activation and feedback
Enable omnichannel events, interactions, communications and engagement
Omnichannel orchestration for activation of audience/user insights
Journey optimisation through integrating quantitative and qualitative feedback
Some ideas on what this could look like taking a product development point of view
V0 Features
1. Data Ingestion Module:
Responsible for collecting, importing, and processing data from various sources (structured and unstructured) for further use in the system.
Features:
Real-time streaming data ingestion.
Batch data ingestion.
Connectors to various data sources (databases, APIs, files, etc.).
2. Data Processing Module:
This would include ETL (Extract, Transform, Load) tools and other data processing capabilities.
Features:
Parallel data processing.
Automated ETL pipeline creation.
Data transformation tools (aggregation, filtering, etc.).
3. Integration Module:
APIs and other interfaces to integrate with other enterprise systems (e.g., CRM, ERP, etc.).
Features:
RESTful APIs for integration with other systems.
Pre-built connectors to common enterprise software (SAP, Salesforce, etc.).
Data export capabilities in various formats (CSV, Excel, JSON, etc.).
4. Real-time Analytics Module:
This provides capabilities for streaming data processing and real-time analytics.
Features:
Real-time data processing capabilities.
Stream analytics for processing and analysing data in real-time
Alerting and notification features for real-time insights.
5. Data Storage and Management Module:
Might include relational databases, data warehouses, data lakes, and other storage systems. Responsible for the organization, management, and retrieval of data.
Features:
Data indexing and search.
Data compression and archiving.
Automatic data lifecycle management.
6. Data Quality Module:
A module to ensure data quality, including data cleaning and anomaly detection.
Features:
Data deduplication.
Outlier detection and handling.
Automated data validation and correction.
7. Metadata Management Module:
This would manage data about your data, helping users understand data origin, transformations it has undergone, and its fitness for specific purposes.
Features:
Automatic metadata extraction and indexing.
User-friendly metadata search interface.
Tools for tracking data lineage (history of data transformations and flows).
8. Data Privacy Module:
This would manage user consent, anonymization of data, and other features to protect the privacy of personal data.
Features:
Data anonymization and pseudonymization tools.
User consent management features.
Compliance checking and reporting for data privacy regulations.
9. Data Governance Module:
This ensures data is used consistently across an organization, data access meets compliance standards, and the overall management of data assets aligns with corporate policies.
Features:
Data classification and policy management tools.
Compliance tracking and reporting features.
Tools for managing data access rights and permissions.
10. Data Cataloguing Module:
This provides a searchable service for users to understand what data is available, its relevance, and how to access it.
Features:
Data tagging and annotation capabilities.
User-friendly search and discovery interfaces.
Tools for automatically updating the catalogue when new data is added.
11. User Interface (UI) / User Experience (UX) Module:
The front-end interface for users to interact with the system, visualize data, and understand results.
Features:
Drag-and-drop interfaces for building analytics workflows.
Customizable user profiles and settings.
Interactive data visualizations.
12. Security Module:
Ensures secure access to data and analytics results, and compliance with data privacy regulations.
Features:
Role-based access control.
Data encryption, both at rest and in transit.
Compliance checking and reporting (for GDPR, CCPA, etc.).
13. Audit and Log Management Module:
To ensure full traceability of actions taken within the platform, there might be a need for a detailed auditing and logging system.
Features:
Detailed activity logs for all user actions.
Audit reports for compliance purposes.
Alerting and notification for unusual or unauthorized activities.
V1 Features
1. Descriptive Analytics Module:
Tools for data exploration, reporting, visualization, and historical data analysis.
Features:
Customizable dashboards and reports.
Data visualization tools (graphs, charts, etc.).
Data aggregation and summarization tools.
2. Diagnostic Analytics Module:
Capabilities for drill-down analysis, data discovery, and cause-effect analysis.
Features:
Drill-down and roll-up capabilities.
Correlation analysis tools.
Cause-effect analysis tools.
3. Predictive Analytics Module:
This would include statistical modelling and machine learning capabilities, trend analysis, and forecasting tools.
Features:
Machine learning model training and evaluation.
Automated feature selection and feature engineering.
Time series forecasting.
4. Prescriptive Analytics Module:
Incorporates decision optimization algorithms, recommendation engines, and simulation tools.
Features:
Decision optimization algorithms.
What-if scenario analysis.
Recommendation engines for personalized suggestions.
Each of these features would contribute to a powerful and flexible analytics platform that can handle a wide range of analytics tasks.
V2 version features
1. Automated Workflow Module:
System for automating analytics workflows, scheduling tasks, and orchestrating complex analytics processes.
Features:
Scheduling and job management capabilities.
Alerting and notification features for job statuses.
Ability to create complex workflow chains with dependencies.
2. Machine Learning Operations (MLOps) Module:
As machine learning becomes more pervasive in analytics, an MLOps module to manage the lifecycle of ML models, from development to deployment and monitoring, would be valuable. This could be a stand-alone or sub-module. Depending on architectural perspective taken.
Features:
Tools for versioning and tracking machine learning models.
Automated model deployment and scaling.
Model performance monitoring and alerting.
3. Collaboration Module:
A feature set that allows users to share findings, notes, reports and work together on data-driven projects.
Features:
Shared workspaces for team-based projects.
Tools for annotating and discussing reports, visualizations, etc.
Real-time or asynchronous collaboration tools.
4. Edge Analytics Module:
If your use case involves IoT devices, this would process data on the device ("at the edge") to reduce the amount of data that needs to be transmitted and allow for faster responses.
Features:
Tools for deploying and managing analytics workloads on edge devices.
Data reduction techniques to minimize the amount of data transmitted from the edge.
Real-time analytics features for processing data at the edge.
These are just some of the many features that could be part of these modules.
Again, the need for these additional features would depend on the specific requirements of the organization and the intended users of the analytics platform. But considering these options could help to create a more powerful, user-friendly, and flexible analytics platform.
V3 version features
The features listed cover a lot of ground, but given the rapid advancement in data science and analytics, there's always a possibility of additional aspects to consider. Here are a few more:
1. AI Explainability Module:
As AI and machine learning models become increasingly complex, it's important to have features that help users understand and interpret these models. This could be integrated into the predictive/prescriptive analytics or MLOps modules.
Features:
Model interpretation tools (like LIME or SHAP) to understand feature influence.
Decision tree visualization for machine learning model decisions.
Text explanations for AI model decisions.
2. AutoML Module:
AutoML tools automate many of the tasks involved in machine learning, such as feature selection, model selection, and hyperparameter tuning. This can make predictive analytics more accessible to non-expert users.
Features:
Automatic feature selection and generation.
Automated machine learning model selection and hyperparameter tuning.
Automated model validation and performance assessment.
3. Data Virtualization Module:
Data virtualization tools create an abstraction layer that allows users to access and manipulate data without needing to know its location or format. This can help to integrate disparate data sources and simplify data management.
Features:
Real-time access to data from disparate sources without physical data movement.
Data federation to allow querying over multiple data sources as if they were a single source.
Data abstraction to provide a unified, business-friendly semantic layer.
4. Embedded Analytics Module:
These allow analytics tools to be integrated directly into other enterprise software, enabling users to access insights without leaving their usual work interfaces.
Features:
APIs and SDKs to integrate analytics into other software.
Customizable, interactive dashboards that can be embedded in other applications.
Real-time updates and notifications within embedded analytics.
5. Natural Language Processing (NLP) Module:
NLP capabilities could allow users to interact with the platform using natural language queries, or could be used to analyse text data within the platform.
Features:
Natural language queries to interact with the platform.
Sentiment analysis on text data.
Text classification, named entity recognition, and other text analytics tasks
6. Advanced Visualization Module:
While basic visualization features were covered under the descriptive analytics module, more advanced capabilities such as 3D visualization, geospatial visualization, or real-time updating visualizations could be beneficial.
Features:
Dynamic 3D data visualizations.
Geospatial data visualization with map overlays.
Real-time updating visualizations for streaming data.
7. Marketplace for Analytics Apps Module:
Similar to app stores, some platforms offer a marketplace where users can find and share custom analytics applications, models, or workflows.
Features:
Platform for users to share and download custom analytics apps.
User ratings and reviews for shared apps.
Tools for developing and uploading custom analytics apps.
These additional features could enhance the capabilities of an analytics platform, making it more versatile and user-friendly. However, it's important to balance the addition of new features with the complexity and usability of the platform. Too many features can overwhelm users and make the platform difficult to navigate and use effectively.
V4 version features
I am sure AutoGPT or some variant could develop new features we don’t even know we need.
Postscript
Before you go, please could you do the following?
Subscribe
Share
Survey
Star
If you got value from reading the article a star liking would be highly appreciated!