These common issues can torpedo your reporting and analytics initiatives. Make your data work harder and generate higher-quality insights by avoiding these key mistakes.
1. Analysis Without Objectives
Without a defined set of business objectives, your analytics efforts are going to struggle. Before diving into your data blindly, ensure that you have gone through a structured process to:
- Identify the business objectives that your program/site/campaign needs to accomplish.
- Translate these business objectives into a quantifiable set of primary KPIs that can clearly evaluate success or failure.
- Ensure coverage across the entire customer journey and lifecycle.
- Set targets for each KPI.
- Support each KPI with secondary/contextual metrics that can explain why a goal is attained or missed.
The result of this exercise will provide a framework that orients all of your analysis to align with business outcomes.
2. Simpson’s Paradox
An overall trend can be deceptive. Consider the following results that were recently brought to us by a digital marketing manager after a creative refresh of their paid social ads:
The creative refresh had been heralded a great success with an impressive 18% increase in click-through rate – but two mistakes had been made.
The first mistake was not conducting a randomized A/B test. The second mistake was that the data had been completely misread, with the marketing manager falling prey to Simpson’s Paradox.
Examining the data again, broken out by audience and layering in the impressions per audience, generates a much different insight:
From this view, we can clearly see that the new creative underperformed in every audience and that the top-line lift was due to a dramatic shift in impressions across these audiences.
Make sure that you don’t misread that top-line trend and always look at your key data segments for context.
3. Correlation Does Not Equal Causation
One of the data community’s most-used maxims – and for good reason. Understanding what is the driving cause of consumer behavior is a goal of almost any marketing or customer analytics practice. To keep it easy, we follow a simple process at Definition 6:
Generate hypotheses from correlation, and test to conclude causation.
A well-designed and randomized A/B/n test is one of the safest methods of taking insights to the next level.
Image from xkcd
4. Overgeneralizing
The human mind is an amazing tool. One of its more remarkable gifts is the ability to recognize patterns and apply those patterns to future scenarios. In data analysis, this can be both a gift and a curse. Our fundamental need to find patterns and apply them can lead to the misuse of data.
For example: Take an A/B test that is run in a single market or geographic area, such as a city. If a winner is declared in that city, does that mean it will win nationally? Too often this type of result is overgeneralized and acted on with unexpected outcomes.
In this case, an expanded test that touches on a diverse set of markets could provide a much more powerful and generalizable test result.
With every insight your analytics program makes, they should also be asking themselves a question: How generalizable is this insight?
5. Relying on a single data source
Messy data has made a victim of us all at one point or another. Defend your business against catastrophic data failure by comparing insights and results from multiple data sources.
This is easier said than done, of course – but it still should be part of your measurement strategy. Periodic checks and balances of your data are vital to ensuring data quality and hygiene remains at a high level.
6. Not setting targets
Setting targets for each one of your KPIs is a crucial exercise that ensures 2 things:
- There is a predefined understanding of what success looks like
- Insights can be structured around “why” success or failure has occurred
This simple exercise can radically change the conversation in your reviews with business leaders. Setting targets can end the time wasted discussing “are we succeeding?” and focus on what is driving success.
Data is a complicated and endlessly nuanced subject – while avoiding these six mistakes is a good starting point, there is always more to learn.
